Comment by rjtc

I am really confused on how it can saturate ARC-AGI but still perform poorly on aggregated benchmarks:

https://artificialanalysis.ai/models

Perhaps if it was allowed this custom harness for all benchmarks it would similarily saturate?