nmitchko

The goal of this auxiliary module is to have the model recurse pre-emit trained on good thinking traces. This includes a 6-to-1 compression of thinking tokens.

Therefore output tokens are decodeable, but are trained compressed. So they are approximations of faster thinking.

Interpretability is a mixed bag even with trained tools on top of existing models.

Fair criticism, i’ll rewrite it… as this is a pet project and post based on another HN comment.

After vLLM startup I observed this on first request only then all further conversations were fine, new and original. Can’t figure out why but seems to be fine after that.

My main motivation is to be able to tune the thinking effort without polluting token IO. Inspiration came from the CoLaR paper and other works showing that repeating laters in an LLM got better results.

I am on my own GPUs so thought it be cool to reduce token output. you can charge per latent loop as a token i guess.

I wanted a knob to keep thinking going until it was sure it was done.

I can do that, main difference seems to be this ablation seems to over think simple prompts, but can do a proper eval (might take 2-3 days)

Fair critique, ill rewrite (this is just my personal pet project but thought was cool to share here)

It’s an adaptation of CoLaR, but my implementation is a little different: - Dedicated stop head to fire when latent thinking hits threshold - MTP support with training taking draft support as first class - different architectural layer 35 -> layer 42 writeback. So latents skip roughly 6.2 tokens of reasoning per token, then never make it to decoded output

Author of this model - I was curious how large models do adaptive thinking and it got me down the path of implementing a model with latent only thinking.

The part I think is most interesting is that the reasoning never touches your context window. The model thinks in a compressed 1024-dim latent space, then decodes just the answer, so a ~6-token reasoning step collapses into one latent step. It also self-terminates: a learned stop head ends the thinking when it's done, so easy questions budget a few steps and hard ones run up to ~256.

The tradeoffs: Blackwell-only (NVFP4), on a vLLM fork (upstream can't serve it due to hash-MoE routing), the trace isn't interpretable, and eval is BBH-only. Happy to answer questions about the closed-loop runtime:: getting the latent injection onto the cudagraph fast path was the fiddly part.