"We absolutely understand why it works, if we didn't understand we would not be able to improve these systems" - Understanding how to improve something does not require understanding how it works.
Growing a plant does not imply understanding soil minerals or photosynthesis.
Training a dog does not imply understanding it's brain.
Using LLMs to improve a codebase does not imply understanding C++.
Improving a PC with new parts, does not imply understanding RAM.
You can understand abstractions about the system, without understanding the system.
Without gradient descent or other forms of automatic optimisation, do you think humans could build something as capable as GPT-1?
Growing a plant does not imply understanding soil minerals or photosynthesis. Training a dog does not imply understanding it's brain. Using LLMs to improve a codebase does not imply understanding C++. Improving a PC with new parts, does not imply understanding RAM.
You can understand abstractions about the system, without understanding the system. Without gradient descent or other forms of automatic optimisation, do you think humans could build something as capable as GPT-1?