Modeling Tips


Focus on solving a problem that fits you.
  • Success comes from creating value, and value is created by solving problems. To solve a problem, you need 1) the ability to solve it and 2) the persistence to stick with it throughout all the difficulties that will surely arise (otherwise the problem would already have been solved).
  • The ability to solve a problem is a combination of domain knowledge and technical expertise -- domain knowledge to envision a feasible solution, and technical expertise to make that vision a reality.
  • The willingness to persist has a more emotional root -- doing something you love, or wanting to fix something that angers you (ideally both).

Consider the full problem from the beginning.
  • A model is worthless if it does not solve the desired problem. Even if it's elegant and theoretically interesting -- if it doesn't actually solve a problem, then it's worthless.
  • It is easier to simplify a convoluted algorithm that solves the problem, than to extend an elegant algorithm that does not solve the problem.
  • Models that are intentionally designed to solve solve specific real problems usually turn out to be theoretically interesting and fun to build, but a theoretically interesting and fun-to-build model designed in a vacuum will rarely happen to solve any sort of real problem. (Looking at you, topological data analysis!)

Both you and your model need to understand the full context surrounding the problem.
  • The first step to building a model is to gather domain knowledge and fully grasp the context in which the model is meant to exist. If you skip this step, then your model might work in theory but probably not in real life.
  • In order to gather domain knowledge, you need to engage in hands-on experience. So, avoid domains where you're averse to doing things manually and getting your hands dirty.
  • A model can only be as good as the underlying data. If you want your model to do what an expert does, it needs to have all the information that an expert uses during their decision-making process. (Heard this one from Jason Roberts, who heard it from Peter Stone.)
  • Better data (or better data preparation) usually adds a lot more alpha than more sophisticated models.

Choose the right level for your first principles.
  • It is often more efficient to manually encode expert knowledge in a structured data set and build a model on top of that, than to attempt to build a model that does everything from scratch.

Leverage your intuition and emotions.
  • Routinely step back from the theory and implementation and observe your model's behavior. It needs to make sense intuitively and "feel" right emotionally. (If you've spent enough time building domain knowledge by doing things manually and getting your hands dirty, then you should have emotional reactions to the decisions the model makes.)
  • The best machine learning model you have is your brain, and your brain only interfaces with interpretable computer models.
  • The more linear and low-dimensional a model is, the easier it is to find good parameters using your intuition alone.
  • Emotion is an essential part of the feedback loop for improving a model: 1) inspect the model's output, 2) produce a negative emotional reaction, 3) introspect your emotions to identify the root cause of the negativity, 4) describe what the output needs to look like order to produce a positive emotional reaction, 5) tweak the model to give the desired output, 6) return to step 1.

Make your model robust.
  • Make your model robust to data issues (but make sure it logs a warning whenever it comes across a data issue). Data issues will happen from time to time, especially if the model is being developed in parallel with the underlying data infrastructure. The model can't just fall over and refuse to work whenever data issues happen.

Never stall out.
  • Keep forward momentum. If a model is not producing a desired behavior and you're out of ideas, then temporarily hard-code the desired behavior as an "intervention", move on, and periodically revisit the intervention to try out more elegant ideas.

Invest in explainability and digestable logs.
  • To gain confidence in your model and speed up the debugging process, it helps to generate human-readable justifications for why your model makes decisions it does.
  • Even if it takes a while, it's often worth organizing your model logs to be highly informative yet easy to skim. Tuning and debugging go much faster if you can see the forest for the trees.

Control the data-generating process.
  • If you don't control the data-generating process, then it becomes vastly more difficult (and sometimes impossible) to resolve data issues.
  • You either need to own the data-generating process yourself or have trust, a good relationship, an open line of communication with the person who does.

Miscellaneous
  • If your desired outcome feels magical, then you probably don't (yet) have enough technical knowledge to achieve it.