nri

You are right, data/concept drift affects both approaches.

What I meant to say was that retraining your model might not fix that if you baked strong assumptions into it. I edited the blog post to make it clearer.

Interesting, that advice is exactly the opposite of the common wisdom. You mean overfitting as in aggressively maximizing your crossvalidation scores? How do you decide for which problems that is a good approach and for which a more conservative approach is better?

Glad you like the post.

I strongly agree with all of your points, especially custom loss functions can be a great tool. If the problem you are trying to solve has some grounding in e.g. physics you can even go a step further and let the model itself mirror the physical equations.

It's like you say, of course these big models are really cool, but I feel like most of the popular machine learning online courses are too narrowly focused on them and many people discard useful techniques if they are not popular in kaggle competitions.