szmerdi

I like the initiative and the large scale collaboration but in terms of quality, my observation on one of the newly added languages (an underrepresented one) is it outputs dramatically grammatically gibberish! As though it hasn't been through the SFT/RLHF step at all.

(P.S. I'm a native speaker of this language. I tested Aya on reasoning, creativity, and summarization with two examples per task)

Each year, national math olympiad committees submit their toughest problems to the IMO. A select group makes the "shortlist," a pool of ~32 questions (8 per geometry, algebra, number theory, combinatorics). From this pool, the final six IMO problems are chosen through a rigorous voting process.

What makes a great IMO problem? It's all about originality. Ideal problems aren't just about applying advanced theorems, but rather test creative thinking and mastery of fundamental principles. Of course, there were cases in which the IMO committee didn't realize a problem was an obvious corrolary of a grad-level theorem, and some well-read (highschool) students benefited from knowing that!

My Journey: I was once a top-ranked competitor myself. After high school, I joined my country's math olympiad committee.

As a former problem designer for IMO and similar contests, I deeply enjoyed reading this paper. At the same time, I'd like to point out that it was clear Geometry had to be the first topic to give up against AI (i.e., smart knowledge and inference-method indexing) Among math olympiad topics, Geometry problems are often the most "mechanical." Once you can express the problem in terms of coordinates (think XY or complex plane) you're looking at a finite set of steps a computer can use to find a solution. Of course, time limits and human error get in the way of this being practical during the IMO itself. Back in the day, I'd use WolframAlpha to verify geometry proofs for problems I designed (+conjectures) using this approach.

Algebra (especially inequalities) can be similar – pull off some intense calculations and you often have your answer.

Where I'm really excited to see intelligent systems make progress is with Number Theory and Combinatorics problems. The search spaces are far more complex and they often require proving that something is impossible. These are the problems that would be difficult to solve with brute force computation.

You're right - we don't know yet but, as mentioned in other replies, 20B is a reasonable guestimate that would make the point about unfair comparison.

Good initiative but it's not a fair comparison IMO. The Gemini Ultra is the one that should've been compared to GPT-4. And, yes, I know it's not available behind an API yet. But does it justify comparing a < 20 billion params model to a model possibly 50x-100x larger without even mentioning this evident gap in their analysis? In the light of scaling law, I believe no.