I’m not as inclined to grade Gary Marcus’ predictions leniently as Gary Marcus seems to be.
• 7-10 GPT-4 level models
He only gets to claim this is true if you count a bunch of different versions of the same base model, or if you’re willing to say that some models that outperform GPT-4 on some benchmarks count as being GPT-4 level. I don’t think Marcus was right in spirit, here.
• No massive advance (no GPT-5, or disappointing GPT-5)
Seems too unquantifiable to judge, I would call o1 a massive advance over 4o, but I’m sure Marcus would not.
• Price wars
I guess so? From what I’ve read, the frontier models companies are still profitable, and OpenAI now has a $200/mo commercial model, hardly the action of a company deciding its prices purely to undercut the competition.
• Very little moat for anyone
It still seems like the only companies who have pulled off frontier model capabilities have spent many millions of dollars doing it. I think this might become true next year but I don’t think this can be judged as correct based on what we saw in 2024 alone.
• No robust solution to hallucinations
You only use words like “robust” in a prediction like this so you have room to weasel out of it later when the hallucinations diminish greatly but don’t quite go extinct.
• Modest lasting corporate adoption
My industry is oil and gas. A pretty hidebound and conservative industry. Adoption of LLMs has been massive.
• Modest profits, split 7-10 ways
Define modest.
I score Marcus at 0/7, at best 2/7.
• 7-10 GPT-4 level models
He only gets to claim this is true if you count a bunch of different versions of the same base model, or if you’re willing to say that some models that outperform GPT-4 on some benchmarks count as being GPT-4 level. I don’t think Marcus was right in spirit, here.
• No massive advance (no GPT-5, or disappointing GPT-5)
Seems too unquantifiable to judge, I would call o1 a massive advance over 4o, but I’m sure Marcus would not.
• Price wars
I guess so? From what I’ve read, the frontier models companies are still profitable, and OpenAI now has a $200/mo commercial model, hardly the action of a company deciding its prices purely to undercut the competition.
• Very little moat for anyone
It still seems like the only companies who have pulled off frontier model capabilities have spent many millions of dollars doing it. I think this might become true next year but I don’t think this can be judged as correct based on what we saw in 2024 alone.
• No robust solution to hallucinations
You only use words like “robust” in a prediction like this so you have room to weasel out of it later when the hallucinations diminish greatly but don’t quite go extinct.
• Modest lasting corporate adoption
My industry is oil and gas. A pretty hidebound and conservative industry. Adoption of LLMs has been massive.
• Modest profits, split 7-10 ways
Define modest.
I score Marcus at 0/7, at best 2/7.