Yes! I have struggled with this concern for a while now.
I think that humans are the ultimate arbiters of quality for humans.
Nothing that is non-human can make that determination because anything that is not human (an LLM) is at best just a really good model. Since all models are necessarily wrong by definition, they can never get it right all the time. When determining what’s good for humans, it is only us that can figure that out.
I also came back to ZAMM when wrestling this question. There must be something there if we have all independently coming to similar conclusions.
I think that humans are the ultimate arbiters of quality for humans.
Nothing that is non-human can make that determination because anything that is not human (an LLM) is at best just a really good model. Since all models are necessarily wrong by definition, they can never get it right all the time. When determining what’s good for humans, it is only us that can figure that out.
I also came back to ZAMM when wrestling this question. There must be something there if we have all independently coming to similar conclusions.
As an aside, this is a true meme.