The last time I was reminded of the bitter lesson was when I read about Guidance & Control Networks, after seeing them used in an autonomous drone that beat the best human FPV pilots [0]. Basically it's using a small MLP (Multi Layer Perceptron) on the order of 200 parameters, and using the drone's state as input and controlling the motors directly with the output. We have all kinds of fancy control theory like MCP (Model Predictive Control), but it turns out that the best solution might be to train a relatively tiny NN using a mix of simulation and collected sensor data instead. It's not better because of huge computation resources, it's actually more computationally efficient than some classic alternatives, but it is more general.
[0] https://www.tudelft.nl/en/2025/lr/autonomous-drone-from-tu-d...
https://www.nature.com/articles/s41586-023-06419-4
https://arxiv.org/abs/2305.13078
https://arxiv.org/abs/2305.02705