Medieval astronomers were able to predict the orbits of planets surprisingly accurately, within 1/10th of a degree, even though they were assuming the Earth was the center of the solar system. They invented a very complicated system of deferents and epicycles (circles within circles). It was completely wrong but the outputs were surprisingly close to reality.
If we were using LLMs to analyze astronomy, we might just get increasingly complicated epicycles and never realize that the Sun is the real center of the solar system. We then never learn about the anomalies in Mercury's orbit that led to the theory of relativity.
So I agree. I seriously question how valuable LLMs can be in science/math beyond working as advanced search functions.
If we were using LLMs to analyze astronomy, we might just get increasingly complicated epicycles and never realize that the Sun is the real center of the solar system. We then never learn about the anomalies in Mercury's orbit that led to the theory of relativity.
So I agree. I seriously question how valuable LLMs can be in science/math beyond working as advanced search functions.