Even if this were all true, it points to a fundamental risk of using LLM's for important tasks, which is that it is not at all clear to a user that this prompt would cause a problem. The LLM doesn't say "I'm sorry Dave, I just can't do that", it just complies with it and gets the wrong answer.
You can always make excuses for the LLM afterwards, but software with hidden risks like this would not be considered good or reliable in any other context.
You can always make excuses for the LLM afterwards, but software with hidden risks like this would not be considered good or reliable in any other context.