I strongly agree with all of your points, especially custom loss functions can be a great tool. If the problem you are trying to solve has some grounding in e.g. physics you can even go a step further and let the model itself mirror the physical equations.
It's like you say, of course these big models are really cool, but I feel like most of the popular machine learning online courses are too narrowly focused on them and many people discard useful techniques if they are not popular in kaggle competitions.
I strongly agree with all of your points, especially custom loss functions can be a great tool. If the problem you are trying to solve has some grounding in e.g. physics you can even go a step further and let the model itself mirror the physical equations.
It's like you say, of course these big models are really cool, but I feel like most of the popular machine learning online courses are too narrowly focused on them and many people discard useful techniques if they are not popular in kaggle competitions.