Modeling Tips
  • The best machine learning model you have is your brain, and your brain only interfaces with interpretable computer models.
  • Keep forward momentum. If a model is not producing a desired behavior and you're out of ideas, then temporarily hard-code the desired behavior as an "intervention", move on, and periodically revisit the intervention to try out more elegant ideas.
  • Emotion is an essential part of the feedback loop for improving a model: 1) inspect the model's output, 2) produce a negative emotional reaction, 3) introspect your emotions to identify the root cause of the negativity, 4) describe what the output needs to look like order to produce a positive emotional reaction, 5) tweak the model to give the desired output, 6) return to step 1.
  • It is easier to simplify a convoluted algorithm that solves the problem, than to extend an elegant algorithm that does not solve the problem.
  • The more linear and low-dimensional a model is, the easier it is to find good parameters using your intuition alone.
  • Even if it takes a while, it's often worth organizing your model logs to be highly informative yet easy to skim. Tuning and debugging go much faster if you can see the forest for the trees.
  • The first step to building a model is to gather domain knowledge and fully grasp the context in which the model is meant to exist. If you skip this step, then your model might work in theory but probably not in real life.
  • If your desired outcome feels magical, then you probably don't (yet) have enough technical knowledge to achieve it.
  • Start with the problem, and work backwards to the model (not the other way around). Models that solve real problems are usually theoretically interesting, but theoretically interesting models rarely solve real problems. (Looking at you, topological data analysis!)
  • A model is only as good as the underlying data. If you want your model to do what an expert does, it needs to have all the information that an expert uses during their decision-making process.
  • To gain more confidence in your model, it helps to generate human-readable justifications for why your model makes decisions it does.

  • "Everybody wants to be a bodybuilder, but nobody wants to lift no heavy-ass weights." -Ronnie Coleman
  • "If you do what you love, you'll never work a day in your life." -Marc Anthony
  • "It is difficult to get a man to understand something, when his salary depends on his not understanding it." -Upton Sinclair

  • To retain information, you must engage in spaced repetition. (book)
  • Students who actively work out problems (as opposed to listening to lectures) learn more, but they feel like they learn less. (paper)