evilmathkid

well not if its an open textbook exam

in non metaphor terms: In many ML situations you can carry the train set with you test time. Eg: KNN, SVM, replay buffers, etc

this is one such case

--

the separate overfitting concern is fair, look at private holdout performance for that. it performs on par with TRM (a comparable model) in the private set, ofc with far less compute

Yeah agree with pg completely. Multiple people I know did this

Yeah but I posted this work back in Dec-Feb lol

no, most models on kaggle are finetuning during test time, (including LLM based approaches)

Pure frontier LLMs dont, but thats because nobody knows how to make it work cleanly and at scale. Once someone makes it work, it will be deployed

I didn't phrase that well and cant edit, so clarifying:

What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"

2 different things are being called leakage here

1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training

2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser compute

--

also re LLMs: they do not follow this 3 split since (a) Incredibly hard to keep a pretrain dataset clean, (b) common in labs to benchmaxx during postraining (and known to do so on ARC)

haha thanks!

As of today yes I agree with you

in the future, for general perf, I am optimistic that someone will figure out an alphazero like approach (ilya/silver/sutton/carmack seem to be working on something like this)

train from scratch only during the 12 hours allowed on Kaggle

Other competitions have implemented things like this before. Eg: OpenAI's Parameter Golf and Keller Jordan's Modded NanoGPT Speedrun

Glad to know you like it!

1) Unfortunately I didn't. I was v new to ML when I did this and didnt have time or skill to try many things. Will try them when I get some time!

2) Possibly, but it would require significant changes and effort. But much larger models would be required imo (must have capacity greater than the complexity of the problem)

> You mean for a specific usecase?

Yepp

> Also aren't frontier models trained 'from scratch'?

The full sentence was supposed to be "training from scratch only on ARC data"

the point was you dont need large scale pretraining

same! I'd often learn during the exam by solving an easier problem and then that would let me tackle a hard problem