barfbagginus

I am leery about how they sweep the documentation use case under the implied rug of "just put the docs and your ide together on the same tiny laptop screen".

This makes even less sense in visual arts work, where having a second monitor for reference materials is always helpful.

My ide often needs to show multiple modules and a console, or else I'm constantly context shifting. So I benefit from a larger main canvas. Likewise, I also need to show API docs. Maybe two or three sets of them. This is like my palette and tool board.

I need serious space, because my working memory can only hold 2 or 3 facts. When I'm working in a serious context, I need to wrangle maybe 20 different facts - say 5 facts from one API doc, 5 facts from another, and so on. It helps to have all the references displayed at once. That way, when I'm working on a detail, I can easily pick up a little color - an important fact I forgot - just by looking at the pallete.

If I'm working on a smaller screen, API docs and IDE become too crowded. I must scroll to find key facts, or jump between windows and tabs. In each case, context thrashing increases. I actually forget things while context switching, leading to extra jumps to recall something that was just on screen a second ago. As a result I can't get as many facts into my brain, the work is very slow, it's not fun, and I'm likely to quit in favor of something more engaging and fun.

My diagnosis? The author understands the bane of context shifting and distractions, but they might not experience the struggle of tackling complex contexts with exceptionally poor working memory, and needing cleanly organized tools and reference materials in order to enter a fun and fully absorbed flow state.

In order to help me streamline my workflow and feel less of a need for a second screen, I invite the author to demonstrate how they track five or six complex documents on a small screen, without making the view ports so small that it incurs constant scrolling, window focusing, and other context thrashing. Failing that, I think the author can be more accommodating of other people's work styles and challenges.

You can see algorithmic aversion every day on HN, in the form of those who complain about the algorithmic portion of the technical interview.

They act as if the only way to get algorithmic knowledge is to grind specific examples, when really if they would read through Knuth they'd be fine.

These are people who see algorithmic knowledge as the bane of their careers rather than seeing algorithms as:

1) a great way to add value to resource constrained projects

2) a trivially simple and easy way to signal programming abilities, letting you easily breeze through the interview

I seriously would hate having to work with one of those who take pride in how little computer science they know, and because I run a math and computational geometry heavy organization, I will never hire them. But I would estimate that algorithmically and mathematically averse coders form the majority of coders.

The recent result shows SOTA progress from something as goofy as generating 5000 python programs until 0.06% of them pass the unit tests. We can imagine our own brains having a thousand random subconscious pre thoughts before our consciously registered though is chosen and amplified out of the hallucinatory subconscious noise. We're still at a point where we're making surprising progress from simple feedback loops, external tools and checkers, retries, backtracking, and other bells and whistles to the LLM model. Some of these even look like world models.

So maybe we can cure LLMs of the hallucinatory leprosy just by bathing them about 333 times in the mundane Jordan river of incremental bolt ons and modifications to formulas.

You should be able to think of the LLM as a random hallucination generator then ask yourself "how do I wire ten thousand random hallucination generators together into a brain?" It's almost certain that there's an answer... And it's almost certain that the answer is even going to be very simple in hindsight. Why? Because llms are already more versatile than the most basic components of the brain and we have not yet integrated them in the scale that components are integrated in the brain.

It's very likely that this is what our brains do at the component level - we run a bunch of feedback coupled hallucination generators that, when we're healthy, generates a balanced and generalizing consciousness - a persistent, reality coupled hallucinatory experience that we sense and interpret and work within as the world model. That just emerges from a network of self correcting natural hallucinators. For evidence, consider work in Cortical Columns and the Thousand brains theory. This suggests our brains have about a million Cortical Columns. Each loads up random inaccurate models of the world... And when we do integration and error correction over that, we get a high level conscious overlay. Sounds like what the author of the currently discussed SOTA did, but with far more sophistication. If the simplest most obvious approach to jamming 5,000 llms together into a brain gives us some mileage, then it's likely that more reasoned and intelligent approach could get these things doing feats like the fundamentally error prone components of our own brains can do when working together.

So I see absolutely no reason we couldn't build an analogy of that with llms as the base hallucinator. They are versatile and accurate enough. We could also use online training llms and working memory buffers as the base components of a Jepa model.

It's pretty easy to imagine that a society of 5000 gpt4 hallucinators could, with the right self administered balances and utilities, find the right answers. That's what the author did to win the 50%.

Therefore I propose that for the current generation it's okay to just mash a bunch of hallucinators together and whip them into the truth. We should be able to do it because our brains have to be able to do it. And if you're really smart, you will find a very efficient mathematical decomposition... Or a totally new model. But for every current LLM inability, it's likely to turn out that sequence of simple modifications can solve it. Will probably accrue a large number of such modifications before someone comes along and thinks of an all-new model then does way better, perhaps taking inspirations from the proposed solutions, or perhaps exploring the negative space around those solutions.