Some of this stems from the idea that everything can be reduced to logic, so as long as we solve for all logical states, the end result will be perfect.
Of course, the second you start interacting with the real world and with humans, that doesn't really apply anymore and you have to get to 'good enough' much earlier than a generator would be able to predict.
Even if you steer a model to a very specific goal with an easy to validate end state, it will do things that within the scope and context of the goal a human would not really do. In theory if you put the context of the entirety of a human into a model, you might be able to get much closer, but we don't even have enough power generation on the planet to process that (unless someone has a secret ten orders of magnitude more efficient way of doing that).
Of course, the second you start interacting with the real world and with humans, that doesn't really apply anymore and you have to get to 'good enough' much earlier than a generator would be able to predict.
Even if you steer a model to a very specific goal with an easy to validate end state, it will do things that within the scope and context of the goal a human would not really do. In theory if you put the context of the entirety of a human into a model, you might be able to get much closer, but we don't even have enough power generation on the planet to process that (unless someone has a secret ten orders of magnitude more efficient way of doing that).