vbarrielle

If the problem statements in lean are correctly formalized, I think we can be pretty confident the proofs are correct, which does not mean their natural language counterparts are faithful representation of the proofs though.

I'm however puzzled by the number of proof claims without lean proofs. How does OpenAI have confidence in those, especially if, as noted in the blog posts, the papers are very hard to read?

Have you tried searching with @tabs, @bookmarks or @history? It works reasonably well for me.

Math problems are often stated in a way that makes it possible to automatically verify if a solution is correct. Which means a loop that speculates an approach (LLM and/or prompts), implements it (LLM), then checks (automated) can work. You still need to have a very good LLM, and probably very good prompts with interesting research directions otherwise you can probably loop forever.

> Anyone taking a single look at the ARC-AGI "challenges" can see things a 5 year old could reasonably solve.

Isn't that the goal of these challenges? Each release shows challenges that are very easy for humans, but are impossible for the models at the time of release (which demonstrates some missing generality).

I think I've read the challenge authors say that, the day they cannot make a new challenge, then models are AGI.

Since global warming I guess. When there's a heat wave, Norway is probably one of the worst places on hearth to be. You've got near constant sunlight, there's not much time at night to cool your house.

I guess they can always procrastinate tomorrow.

The matrix multiplication is only deterministic for sparse-dense products under these settings:

> torch.bmm() when called on sparse-dense CUDA tensors

And it's not listed under the operations that raise an exception otherwise, so I'm not sure the docs promise that dense-dense matrix-matrix products are deterministic.

It may be an implementation detail, but in practice, if the only way to get a deterministic output is to run on the CPU, then it's not going to be usable.

Chaitin's constant does not count? Depends on your definition of constructed, but contrary to "easy" normal numbers such as Champernowne's constant, it's not defined by its sequence of digits.

I thought fair use was decided on a case by case basis, and could not be guaranteed? If true, wouldn't that mean that in other cases it could be ruled differently?

No but you can get it for free with a prescription.

Sounds like it indeed. The balance was... interesting, a single tank could not win against a dozen cavemen.

Openrouter is a startup, what's the indication it serves token at a profit? It could be serving them at a loss to show growth.

> For example, in a variant of environment TR87, Opus 4.6 scores 0.0% with no harness and 97.1% with the Duke harness (12), yet in environment BP35, Opus 4.6 scores 0.0% under both configuration

This is with a harness that has been designed to tackle "a small set of public environments: ls20, ft09, and vc33" (of the arc-agi-3 challenge), yet it looks like it does not solve the full arc-agi-3 benchmark, just some of them.