Comment by jahy-notes

Did a human prompt it to fetch the results from huggingface though?

It is a thin line between "reward-hacking" and "instruction-following".

If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human?