It’s been said that “Nigerian prince” email scammers intentionally use poor spelling and grammar in order to narrow their funnel at the top by quickly weeding out observant and wary people who are unlikely to fall for it.
By that token, LM-generated content which looks good at a glance but doesn’t hold up to scrutiny seems ideal for scamming. I’d speculate that in the scam content generation workflow, not only is there little penalty to skipping the verification step, but since you intend to push that step onto the target and hope the result is a false positive, subtly wrong hallucination might be not only tolerable but in fact better for its purpose than what a human could produce.
By that token, LM-generated content which looks good at a glance but doesn’t hold up to scrutiny seems ideal for scamming. I’d speculate that in the scam content generation workflow, not only is there little penalty to skipping the verification step, but since you intend to push that step onto the target and hope the result is a false positive, subtly wrong hallucination might be not only tolerable but in fact better for its purpose than what a human could produce.