My theory is that the LLMs can only think in terms of data structures of a certain "dimensionality".
So for example I tried to have LLMs write xml parsers from scratch in Q3 2025, but they kept getting confused and midstream would migrate from c# to xml (that is their solution was half c# half xml file). So thinking in a tree data structure was beyond them.
Then Q1 2026 they were able to write parsers including parsers for languages where I specified with a bunch of examples.
But I tried specifying a language that I thought would require storing the AST in a graph structure. Everytime the LLM would instead parse into a tree and then do post processing on the tree to get the right output. So perhaps graph structures are (were?) beyond them.
I haven't investigated forth for a while, but it's been on my to-do list. I'll have to bump it up some because it sounds like I can kill two birds with one stone here.
I do kind of wonder if tree space is a local maximum, though. You can do quite a lot of things in that space and most of the code that I've read that leans on some sort of graphical thinking (like heavy usage of global variables that effect the execution path all over the call graph) is a less than ideal substrate to have to think through.
[One of the problems with investigating this is that I can't just ask it to solve a problem that is obviously just dijkstra because that's in the training data.]
So for example I tried to have LLMs write xml parsers from scratch in Q3 2025, but they kept getting confused and midstream would migrate from c# to xml (that is their solution was half c# half xml file). So thinking in a tree data structure was beyond them.
Then Q1 2026 they were able to write parsers including parsers for languages where I specified with a bunch of examples.
But I tried specifying a language that I thought would require storing the AST in a graph structure. Everytime the LLM would instead parse into a tree and then do post processing on the tree to get the right output. So perhaps graph structures are (were?) beyond them.
I haven't investigated forth for a while, but it's been on my to-do list. I'll have to bump it up some because it sounds like I can kill two birds with one stone here.
I do kind of wonder if tree space is a local maximum, though. You can do quite a lot of things in that space and most of the code that I've read that leans on some sort of graphical thinking (like heavy usage of global variables that effect the execution path all over the call graph) is a less than ideal substrate to have to think through.
[One of the problems with investigating this is that I can't just ask it to solve a problem that is obviously just dijkstra because that's in the training data.]