Comment by ChrisRackauckas

There's also data in another source: domain knowledge and physical laws. This is the core tenant of scientific machine learning techniques (SciML). I have a talk that walks through how you can go from something without domain knowledge (like a neural ODE) and how as you add more prior model knowledge to the system the extrapolation accuracy improves even for small amounts of data (https://www.youtube.com/watch?v=FihLyzdjN_8). People have used these techniques in all kinds of places, like extrapolating black hole trajectories and building earthquake-safe buildings, off of like 20-40 data points (https://www.youtube.com/watch?v=eSeY4K4bITI). While there is a lot of work to be done in the domain still, there's a lot of empirical evidence that this approach does indeed decrease the data requirements (but is then no longer "pure" machine learning in some sense)