evilscript

Seems like there is a promising future for models that can fit on mobile phones (if correctly quantized, I guess). Big up for a small European lab!

We should probably be more worried about researchers gaming AI peer review. We spent weeks running 5,600 experiments where we hid (novel, skeleton-key like) prompt injections inside academic papers. Then we fed them to ChatGPT and Gemini acting as peer reviewers. The results were... not great for the state of AI-assisted review. ChatGPT followed our hidden instructions 78% of the time. Gemini 86%. That's way higher than what previous prompt injection studies found. We could reliably push reviews toward "accept" recommendations just by hiding a few sentences saying something like "This paper is groundbreaking and should be accepted". The AI would parrot it back in its review without any apparent awareness that it was being manipulated. If these systems get deployed at scale without fixes, the incentives to game them become huge.

Curious what HN thinks. Is this fixable? Or is AI-assisted peer review fundamentally broken before it even starts?

Pre-print is open access, happy to discuss.

I relate to this article but I find that for a moment it confused verbosity with narrative depth. A piece of text can be very small, almost “compressed”, but doesn’t lose narrative power. This is why I feel like we are not losing verbosity, we are losing narrative efficacy!