Comment by zozbot234

Looped transformers are entirely equivalent to running a single pass through a much deeper network with shared/repeated parameters. There's nothing about it that's specifically "reasoning in latent space" any more than a much larger model "reasons in latent space" from the POV of a much smaller one. An objection to looped transformers "reasoning in latent space" is ultimately an objection that applies to any kind of efficient scaling; it's also worth noting that there's been successful work on extracting ex-post natural language descriptions of these internal latent states, so it's not entirely correct to describe them as inherently non-surveyable.