> Time and time again, when I try some newly released open weights model I am disappointed
The article is correct that the major labs are not benchmaxxing, but I think it overstates their focus being on something else.
They don’t play the benchmark manipulation game because they’re comfortably in the lead and they have their own teams producing internal benchmarks and targets that are representative of what their users expect.
Your experience with the open weights models is not missing anything. Nearly all of them are disappointing if you set your expectations according to where they fall in the benchmark charts. They can be useful for some tasks once you’ve developed an intuition for their capabilities, but you have to start learning their relative cost per task and their limits, which takes a lot of observation and honestly might not be worth it.
One of my unpopular open weight model hot takes is that, right now, OpenAI’s Luna model is often cheaper per task than even DeepSeek and others (paying API prices, not based on the fluctuating subscription prices).
The real advantage of open models has always been that you can run them on your own hardware or a rented server and have more control over where your data goes. Some of the hype about them being 3 months behind frontier models or people making claims that 27B local models have replaced their Claude subscription become extremely questionable if you actually use both open weight and frontier models frequently.
The article is correct that the major labs are not benchmaxxing, but I think it overstates their focus being on something else.
They don’t play the benchmark manipulation game because they’re comfortably in the lead and they have their own teams producing internal benchmarks and targets that are representative of what their users expect.
Your experience with the open weights models is not missing anything. Nearly all of them are disappointing if you set your expectations according to where they fall in the benchmark charts. They can be useful for some tasks once you’ve developed an intuition for their capabilities, but you have to start learning their relative cost per task and their limits, which takes a lot of observation and honestly might not be worth it.
One of my unpopular open weight model hot takes is that, right now, OpenAI’s Luna model is often cheaper per task than even DeepSeek and others (paying API prices, not based on the fluctuating subscription prices).
The real advantage of open models has always been that you can run them on your own hardware or a rented server and have more control over where your data goes. Some of the hype about them being 3 months behind frontier models or people making claims that 27B local models have replaced their Claude subscription become extremely questionable if you actually use both open weight and frontier models frequently.