MattRogish

Yes! I was having Argon summarize some project updates and it... just made up entire paragraphs that had no basis in reality. It felt like I was back in GPT3 days.

Will have to give Carbon a try, but if it is anything like what I saw before it was entirely unusable.

Back in the olden days (2015-ish?) KLM was having a really, really cheap business fare sale JFK-AMS; I snagged it with Delta miles (if I recall correctly) - and flew there and back in their 747 in the upper deck (just to take the flight; didn't have anything to do in AMS). It was really quite nice; it was the first and apparently last time I've taken the 744. I'm really glad I was able to do it.

I calculate* the appropriate overlap and the slicer overlaps a certain amount of the previous slice. There is some post-processing assembly required, but it's trivial.

[*] SWAG line height, trial and error to figure out the right amount of overlap given LLM error rates, etc.

I do OCR of images, and that's exactly what I do. I take one big image and slice it into many smaller ones, and send those to the LLM. Perfect every time, unlike using the whole image which resulted in hot garbage.

I'm not saying they are not trying - I'm saying we're inventing new problems faster than any Lab can:

1) Identify the gaps

2) Determine how to fix them

3) Implement a fix (especially if that fix is: identify and find experts)

4) And judge the result

How do they know [person] is an expert in [some field]? How do they find that person? How many experts are necessary to give the right information? How do we evaluate the results, especially if it's novel?

You can find a lot of people who disagree on many topics, and those turtles go all the way down.

I'm not in disagreement that your work will help reduce hallucinations and improve model performance! It is.

I predict (I hope I'm wrong!) that we're going to hit some asymptote that is not at 0% hallucinations (and I would even put a substantial nonzero probability that "overall" hallucination rate bottoms out at some minimum and then slowly grows because we just can't keep up with the new garbage we throw at it).

I'd have to imagine there are wildly diminishing marginal returns to additional SFT/post-training passes.

There are a bounded number of (useful) derivations/combinations of Duff's device.

If Frontier Labs wish to reduce hallucinations on factual things, they will have to hire people (or the data providers will need to) to do fundamental research above and beyond what is available in extant literature and the web. IE if the LLMs want to lower precision error, they need to go out and actually find more expertise. If the wikipedia page for Pompey lacks data, where are they going to get it from? How would they even _identify_ that the page has holes?

Yes, they can digitize more books but that is untrustworthy data - if there were enough eyeballs on a particular work, it would be in the internet. If it's not, they'd need to hire the experts themselves. They need expert reviewers in virtually every interesting topic, which fundamentally is an intractable problem, especially since things change all the time. Maybe even uninteresting topics, too?

I dunno, it doesn't seem to me "more data" is the magic bullet here. Yeah, it will "help" but we're already on the flat part of the S shaped curve.

My take from trying to understand this stuff is some sort of algorithmic improvement is necessary to get another step change in how well LLMs perform in this area. I could be wrong!

Yes, for fun I tried to make a Mahjong solver and NONE of the SOTA frontier models could understand what they were looking at to determine tile occlusion/geometry to build the DAG.

I had to spoon feed it an algorithm - here's how you determine if a tile is on top of another one, etc. etc.

Anything that involves, well, "3d space" they don't seem to do very well on it at all (which makes sense, of course)

Agreed, my vibes tell me 4.6 is a better coder than 4.7. 4.7 is a much better strategic thinker and maintains overall "better architecture" than 5.5. 5.5 is way better than either at coding, but more expensive. So I have 4.7 do the planning/architecture, 4.6 does the coding, then 5.5 critiques and fixes it.

The other thing I suspect is that "Just give me True/False" cuts off a large amount of the search space a modern-day LLM uses to help it answer questions (you can see it in reasoning traces but the act of writing the explanation helps guide it toward a better answer and gives it better likelihood it backtracks on a bad decision).

If you let it spew out an explanation along with the answer, I'm curious if the accuracy will improve (I suspect it will).

Also, this is why investors and CEOs are so in love with "LLMs are the route to AGI!"

When some rich/powerful person says "I have to go to Davos, figure it out" their workers know so much context that no LLM is going to ever be able to incorporate, because it isn't written down and is idiosyncratic. (Really, though, the assistant will just say "you're going to Davos next week, the helicopter will pick you up at 3p on Friday" but you know..)

The rich person's assistant knows who else is on the corporate jet, and that X doesn't like Y, and so they should take a different plane. Or get a different accommodation. Oh, Person X doesn't like to fly on an empty stomach, so they should eat first, and that changes all sorts of other downstream implications. Oh, your best friend lives in this city, and I know you love to see them, so I'm going to send you a day or two early so you can meet up with them. etc. etc. etc.

The investor dream of "AGI" is modeled off of the army of employees that make investors/ceos/etc lives easier, and there is a nearly insurmountable gap between what LLMs can do, context they can get, and the availability of all of that information. (To me, the magnitude of this investor <> fundamental reality gap is the entirety of the "bubble". I love AI coding, but it's never gonna do the things investors think it can, to justify the crazy valuations)

Thank you; you’re right - context matters and now more than ever there are a ton of folks looking involuntarily. Grace is always needed, but now especially.

Yes! It reminds me of the tabs vs spaces argument. Yeah, you have your preference, but as long as everyone on your team uses the same convention it doesn’t matter

The Rails apps I’ve programmed with LLMs seem to work a LOT better than arbitrary python or ruby or JavaScript apps. I chalk that up to “there are a gazillion examples of omiauth in Rails that the LLM can’t really stray off the path. It just works.”

That means I let the agent do things the way it wants to, not because I have a preference. So we’re using turbo and Hotwire and whatever it is it’s doing. And I’m using React for some other problems. Not because I know React, but because the LLM does.

In golf it is said to “let the club do the work”. Over control leads to disaster. Same with LLMs. Not saying let it do whatever but if there are widely baked in conventions you’ll be far better off letting those do the work.