Comment by willbudd

Will-free? Unless that's a play on my first name, I'm not sure I agree. I see no reason why AI would have any difficulty defining its own reward functions. Especially if it also has an abstract overarching reward function that's wide enough in scope. For example, "learn as much about the universe as you can" would allow a very long curiosity-driven bucket list of pursuits it could "long" for.