dinp

The last time I checked, for the arc agi 3 leaderboard, the models are given a simple prompt and the game input and asked to play the game, no harness/tools. If harnesses were allowed, I would expect the benchmark to be saturated. There were a few harness attempts, but they could only be evaluated on the public set, so it's not an apples to apples comparison.

My guess is, the large score jump for Opus 5 is mainly because of getting the right RL envs for training.

It's becoming harder and more expensive to build and run meaningful benchmarks, it would be interesting to see what they do with arc agi 4, maybe just give it gameboy/steam games and see how they compare vs a human baseline? The latency requirements and very long horizons in games could be an interesting challenge for llms.

Zooming out a little, all the ai companies invested a lot of resources into safety research and guardrails, but none of that prevented a "straightforward" misalignment. I'm not sure how to reconcile this, maybe we shouldn't be so confident in our predictions about the future? I see a lot of discourse along these lines:

- have bold, strong beliefs about how ai is going to evolve

- implicitly assume it's practically guaranteed

- discussions start with this baseline now

About slow take off, fast take off, agi, job loss, curing cancer... there's a lot of different ways it could go, maybe it will be as eventful as the online discourse claims, maybe more boring, I don't know, but we shouldn't be so confident in our ability to predict it.

I thought I was the only person going crazy by the new default behavior not showing the file names! Please don't expect users to understand your product details and config options in such detail, it was working well before, let it remain. Or at least show some message like "to view file names, do xyz" in the ui for a few days after such a change.

While we're here, another thing that's annoying: the token counter. While claude is working, it read some files, makes an edit, let's say token counter is at 2k tokens, I accept the edit, now it starts counting very fast from 0 to 2k and then shows normal inference speed changes to 2.1k, 2.3k etc. So wanted to confirm: is that just some UI decision and not actually using 2k tokens again? If so, it would be nice to have it off, just continue counting where you left off.

Another thing: is it possible to turn off the words like finagling and similar (I can't remember the spelling of any of them) ?

The article mentions, he is going to run the marathon, looking forward to what he can do in that distance. I feel it's only a matter of time until someone breaks the 2 hour barrier in an official race. Lot of people thought it would be Kelvin Kiptum, unfortunately he passed away in an accident.

I think political news are not encouraged here, exceptions for when interesting discussions are possible.

Judging by the quality of comments here and in the linked submissions, it's a good thing.

The impact this organization had was incredible. I doubt they would have been able to do this work if they were based out of any other country, which makes me wonder how the US legal system, regulators and law enforcement in general are not extremely corrupt. What reasons or incentives make the system work in the US? Of course there are many instances of corruption and injustice, but in comparison to almost any other country, it seems to work surprisingly well.

Great work! When I use models like o1, they work better than sonnet and 4o for tasks that require some thinking but the output is often very verbose. Is it possible to get the best of both worlds? The thinking takes place resulting in better performance but the output is straightforward to work with like with sonnet and 4o. Did you observe similar behaviour with the 1B and 3B models? How does the model behaviour change when used for normal tasks that don't require thinking?

Also how well do these models work to extract structured output? Eg- perform ocr on some hand written text with math, convert to html and format formulas correctly etc. Single shot prompting doesn't work well with such problems but splitting the steps into consecutive api calls works well.

The idea of system 1 and system 2 had a profound impact on me. While specific conclusions in the book were reported to be based on low quality data, it doesn't take away from the fact that it gave me a new mental lens to look at things and understand people's behaviour.

Somehow the idea of perpetually paying property taxes and land value taxes doesn't sound appealing to me, especially since businesses already pay taxes. I don't understand the argument of designing a system to hurt a specific business type such as low value businesses. If there's a loophole such as lack of sales tax for car washes, fix that, but let the playing field remain even. If desirable high value businesses aren't able to compete with car washes, isn't that the market doing it's thing? Introducing additional property and land value taxes might discourage low value businesses, but what are the 2nd and 3rd order effects of such a change?