I also witnesses this first hand at a Biotech company I worked at... we were using many variants of machine learning algorithms to develop predictive models of cell culture and separation processes. Problem is... the models have so many parameters in order to get a useful fit that the same model can also fit a carrot or an elephant.
We found that dynamic parameter estimation on ODE/DAE/PDE system models, while harder to develop, actually worked much better and gave us real insight into the processes.
So now my advice is others is "if you can start with some first principles equation or system of equations... start there and use optimization/regression to fit the model to the data."
AND: "if you don't think such equations exist for your problem... read/research more, because some useful equations probably do exist."
This is usually pretty straightforward for engineering and science applications... equations exist or can be derived for the system under study.
In my very limited exposure to other areas of machine learning application... I have found quite a bit of mathematical science related to marketing, human behavior, etc.
So now my advice is others is "if you can start with some first principles equation or system of equations... start there and use optimization/regression to fit the model to the data."
AND: "if you don't think such equations exist for your problem... read/research more, because some useful equations probably do exist."
This is usually pretty straightforward for engineering and science applications... equations exist or can be derived for the system under study.
In my very limited exposure to other areas of machine learning application... I have found quite a bit of mathematical science related to marketing, human behavior, etc.