ClintEhrlich

Hi HN, Here is the technical report for a family of LLMs my co-founder and I built with ten million tokens of context. It's really a proof of concept for a new kind of attention, called VSA (Voltropy Scalable Attention.)

There were previously two approaches to getting context to work at this scale. One was to approximate attention in ways that degraded reasoning-quality at lower context lengths. The other was to abandon the transformer architecture in favor of cheaper alternatives that only did e.g. recurrent computation, like state space models.

We designed VSA to be a third path: One that would extend context by an order of magnitude while also increasing reasoning quality at shorter contexts. And what is striking about the results we share today is that this worked, even when we retrofitted VSA to transformers that were training with conventional attention.

When we augmented DeepSeek v4.0 Flash with VSA, the resulting model scored better at ten million tokens than the original did at one million tokens. To our surprise, it did even better than DeepSeek's next-generation model, v4.1 Flash. In fact, the 1M-token performance was so strong that it beat Anthropic's Fable 5.1 by around 10%.

All this is to say that we believe we've found an important new primitive for building LLMs. As we discuss in the blog post, we think that this could scale way beyond 10M tokens, and that stretching it to billions or trillions of tokens could fundamentally reshape what LLMs are.

First, it could solve continual learning, by making context windows so large that agents can learn continuously within ever leaving a context window. Basically, a loophole to solve continual learning without post-deployment weight updated.

Second, it could allow for much of the data that is currently stored in weights through lossy memorization to be moved into a structured, context-like format. Basically, the goal would be to make LLMs less like human brains and more like classical Von Neumann machines.

Obviously, that vision requires scaling VSA far beyond where it is today. But we're excited to share this first step on the journey with you all.

Hi HN,

This family of LLMs has ten million tokens of native context, but it's really a proof of concept for a new kind of attention, called VSA (Voltropy Scalable Attention.)

There were previously two approaches to getting context to work at this scale. One was to approximate attention in ways that degraded reasoning-quality at lower context lengths. The other was to abandon the transformer architecture in favor of cheaper alternatives that only did e.g. recurrent computation, like state space models.

We designed VSA to be a third path: One that would extend context by an order of magnitude while also increasing reasoning quality at shorter contexts. And what is striking about the results we share today is that this worked, even when we retrofitted VSA to transformers that were training with conventional attention.

When we augmented DeepSeek v4.0 Flash with VSA, the resulting model scored better at ten million tokens than the original did at one million tokens. To our surprise, it did even better than DeepSeek's next-generation model, v4.1 Flash. In fact, the 1M-token performance was so strong that it beat Anthropic's Fable 5.1 by around 10%.

All this is to say that we believe we've found an important new primitive for building LLMs. As we discuss in the blog post, we think that this could scale way beyond 10M tokens, and that stretching it to billions or trillions of tokens could fundamentally reshape what LLMs are.

First, it could solve continual learning, by making context windows so large that agents can learn continuously within ever leaving a context window. Basically, a loophole to solve continual learning without post-deployment weight updated.

Second, it could allow for much of the data that is currently stored in weights through lossy memorization to be moved into a structured, context-like format. Basically, the goal would be to make LLMs less like human brains and more like classical Von Neumann machines.

Obviously, that vision requires scaling VSA far beyond where it is today. But we're excited to share this first step on the journey with you all.

You could definitely build a coding agent that way, and it sounds like you've done it. We store the conversation history because:

1. In our use of coding agents, we find that there are often things referenced earlier in the conversation (API keys, endpoint addresses, feedback to the agent, etc.) that it's useful to have persist.

2. This is a general-purpose LLM memory system, which we've just used here to build a coding agent. But it is also designed for personal assistants, legal LLMs, etc.

By construction, individual summaries are not typically large enough to overload the context window when expanded.

The reason that the volume is potentially arbitrarily large is that one sub-agent can call lcm_expand multiple times - either vertically or horizontally. But that's a process that occurs gradually as the tool is used repeatedly.

This has not been a problem in our testing, but if it were a problem it would be easy to prevent sub-agents from invoking lcm_expand once their context buffer has reached a specified threshold.

Thanks for the kind words! Looks cool!

Hi NWU,

We don't have any other materials yet, but let's see if this lands for you. I can run you through a couple simpler versions of the system, why they don't work, and how that informs our ultimate design.

The most basic part of the system is "two layers". Layer 1 is the "ground truth" of the conversation - the whole text the user sees. Layer 2 is what the model sees, i.e., the active context window.

In a perfect world, those would be the same thing. But, as you know, context lengths aren't long enough for that, so we can't fit everything from Layer 1 into Layer 2.

So instead we keep a "pointer" to the appropriate part of Layer 1 in Layer 2. That pointer takes the form of a summary. But it's not a summary designed to contain all information. It's more like a "label" that makes sure the model knows where to look.

The naive version of the system would allow the main model to expand Layer 2 summaries by importing all of the underlying data from Layer 1. But this doesn't work well, because then you just end up re-filling the Layer 2 context window.

So instead you let the main model clone itself, the clone expands the summary in its context (and can do this for multiple summaries, transforming each into the original uncompressed text), and then the clone returns whatever information the main thread requires.

Where this system would not fully match the capabilities of RLMs is that, by writing a script that calls itself e.g. thousands of times, an RLM has the ability to make many more recursive tool calls than can fit in a context window. So we fix that using operator-level recursion, i.e., we give the LLM a tool, map, that executes arbitrary recursion, without the LLM having to write a custom script to accomplish that.

Hope this helps!

- Clint

Our system uses sub-agents as a core part of its architecture.

That terminology can be confusing, because in other cases (and sometimes in our own architecture, like when executing thousands of operations via MAP) a sub-agent may be a smaller model given less complex individual tasks.

But the core mechanism we use for simulating unlimited context is to allow the main model to spin up instances of itself (sub-agents) with the previously summarized portion of the context expanded into its full, uncompressed state.

Expanding summaries into full text in sub-agents rather than the main thread is a critical part of our architecture, because it prevents the main context window from filling up.

Just passed this on to my co-author who is working on the plug-in. Really appreciate the suggestions!

We will probably ship a fairly basic version to start, but I think there are a lot of cool things that can be added.

Oh and to be clear YES you can try it!!!

Just bring an API key. :)

github.com/voltropy/volt

Thanks for the kind words.

Yes, we think there is a ton of low-hanging fruit from taking lessons from OS/PL theory and applying them to LLM tooling.

This is our first contribution in that direction. There will be more!

Yes, that is actually the next thing we are shipping!

We have heard from a ton of OpenClaw users that the biggest barrier to them getting everything they want out of their agents is that memory is not a solved problem.

LCM could be a great solution to that. Stay tuned -- will ship it ASAP.

Hi, I'm Clint, one of the co-authors of this paper.

I'd like to quickly summarize what is different about our approach and why it matters.

Our work was inspired by brilliant research done at MIT CSAIL on "Recursive Language Models" (RLMs). One of the controversies has been whether these models are just a formalization of what agents like Claude Code already do vs. whether they bring new capabilities to the table.

By outperforming Claude on the major long-context benchmark, we provide a strong signal that something fundamentally new is happening. (In other words, it's not "just Claude Code" because it demonstrably outperforms Claude Code in the long-context regime.)

Where our contribution, LCM, differs from RLMs is how we handle recursion. RLMs use "symbolic recursion" -- i.e., they have an LLM write a script to recursively call itself in order to manipulate the context, which is stored in a REPL. This provides maximum flexibility... but it often goes wrong, since the LLM may write imperfect scripts.

LCM attempts to decompose the recursion from RLMs into deterministic primitives so that the control flow can be managed by an engine rather than left to the whims of the LLM. In practice, this means we replace bespoke scripts with two mechanisms: (1) A DAG-based context management system that works like paged virtual memory, except for managing conversations and files; and (2) Operator-level recursion, like "Map" for LLMs, which lets one tool call process thousands of tasks.

An analogy we draw in the paper is the evolution from GO-TO statements (of Dijkstra's "Considered Harmful" fame) to structured programming. RLMs are maximally expressive, but all of that power comes with the risk of things going awry. We have built a more mechanistic system, which can provide stronger guarantees when deployed in production with today's models.

Happy to answer any questions! Thanks for taking a look at the paper!