Comment

Comment by spijdar

It's worth pointing out that merely having the chain-of-thought in latent space isn't by itself "The Most Forbidden Technique". Rather, it's doing what the authors propose as the next step in the paper:

  Hybrid generation can also make post-training more efficient. Because latent thoughts remain auditable during rollout, an external critic can assign step-level credit to latent spans, branch promising trajectories, or terminate ones whose latent reasoning has already gone wrong, before any text or tool call is produced.
Unless I'm misunderstanding, they're suggesting RL training against the CoT. Which, uh, that is The Most Forbidden Technique, as I understand it, when combined with the latent reasoning space. It allows for the model to "learn"/optimize for malicious behaviors that are just far enough away from the quantized token they're auditing against.
parent spijdar 14 days ago

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton