Comment by yconst

Interesting point. However, I'd like to argue that there is a fine line between the conventional definition of a utility function and the ability of an organism to survive in an environment. The latter is truly open-ended, and not really externally imposed, i.e. different animals/people fare throughout life "optimising" completely different functions (closure?). A quote from a Kozma paper comes to mind: "Intelligence is characterized by the flexible and creative pursuit of endogenously defined goals". I believe this quite well summarises the open-ended nature of the task at hand. As I understand it, indeed, goals, rewards, risks, hazards, they are all in the game and shape the decision-making of the agent, it's "policy". But the way they are formalized for each individual and situation, well, is probably subject to constant redefinition itself.