Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic".
I am surprised this is not already the case.
Edit: or even better "this is not possible because X"
I am surprised this is not already the case.
Edit: or even better "this is not possible because X"