Comment by coke12

How could you possibly know what an LLM would do in that situation? The whole point is they exhibit occasionally-surprising emergent behaviors so that's why people are testing them like this in the first place.