tedsanders

To elaborate, I can ask each model to write a doc about something I know well and tell you which output I like better.

But to assess whether my preference on my problem is going to generalize across other people, other programming languages, other documentation types/formats, etc., and then analyze statistical significance is a thornier question that I don't have the data to answer. I'd rather make no recommendation than a weakly informed recommendation.

The error bars on eval graphs are commonly like ±5%. It's fairly common to get out-of-order points when the gap between them is only a few percentage points. If we ran it again, or ran it with minor alterations, I'd expect the ordering to be expected. In general, we don't massage any of our eval numbers and for good statistical hygiene we'll report the first run we do, even if later runs show slightly smoother curves.

(I work at OpenAI.)

Very fair. There's always a tradeoff in the number of products, and it's hard to make everyone happy. My suspicion is that the optimal number is probably a bit higher for developers on our API (who often build their own internal evals) than for individual users (who don't want to waste time re-optimizing every month). We're excited by Terra's uniquely fast computer use, but if it traction is poor, maybe we'll go back to 3. It all depends on the market signal - we only want to ship models people find uniquely useful.

These days, version number signifies a few things at once: the order of release, the relationship to other releases, and the magnitude of improvement. So something like 6.1 would indicate it was launched after 5.6, may share some similarities to other 6.1 models in terms of training data / personality (which is why it's not 5.7), and isn't a giant enough leap to merit the headlines and expectations that would come with a 7.

Back in the day we used numbers to also sometimes indicate size, which could make it extra confusing: e.g., we launched GPT-4.1 (an incremental update on GPT-4) after GPT-4.5 (a new model that was a big jump over GPT-4).

> Do you then recommend Sol for technical documentation?

I don't know, to be honest. Technical documentation is such a multidimensional skill, and I also don't know your budget. I'd imagine Terra might have the most readable prose, but Astra might be more likely to catch bugs or mistakes to highlight. If the documentation is about a more esoteric language or situation, Astra might have a greater edge than if it's a common language or situation. If you have a long list of documentation guidelines, that might also give more of an edge to Astra. I wouldn't want to make a recommendation without doing a lot of testing, and I haven't done a lot of testing.

I promise you this would become a giant mess. No one wants to pick GPT-good-at-writing-but-not-in-documents-and-decent-at-instruction-following-but-has-some-known-quirks-to-be-aware-of-p.s.-good-at-knock-knock-jokes-MAY-2027.

Even "writing" as a single concept is very multidimensional. There's fiction writing, nonfiction for experts, nonfiction for learners, deep research, quick lookup Q&A, copyediting, writing in documents / artifacts, instruction following about style, and default quirks (e.g., em dashes, negation, loving the word "tapestry"). Some models are great writers but less diverse, meaning people prefer the first story they read but disprefer the tenth. How does one label such a model?

What would you have like to seen instead?

The problem with small, medium, large naming is that they don't last. In a few years our names would have to advance to labels like giant, behemoth, gargantua. This is different than clothing, where we know that in five years humans will be most the same size as today.

As a point of comparison, GPT-3 was launched in a series of small, medium, large, and XL. Back then, XL meant a mere 1.3 billion parameters!

Same reason that Apple doesn't name its series of phones the iPhone cheap, iPhone regular, iPhone semi-expensive, and iPhone expensive. It's poor branding, isn't easily extended to future models, and gets confusing when comparing across model generations, where an iPhone 14 expensive might be cheaper than an iPhone 16 cheap.

I work at OpenAI and I can try to explain our logic.

In general, LLM performance is incredibly multidimensional. There's no guarantee of a fixed ordering where the model that is best at math is also the model that's best at jokes. Despite that complexity, much of the variation in performance is explained by just a single variable: model size. In general, models that are bigger will (a) be more performant / reliable, (b) cost more per token, and (c) have better recall of esoteric facts. As you note, our models are ordered:

Luna < Terra < Sol < Astra

Even today, I'd recommend this as a general starting point for picking models. If you care about quality only, pick Astra. If you're a bit price sensitive, pick Sol. If you're very price sensitive, pick Terra or Luna.

Of course, model size doesn't capture all variation. Models that are trained later might have new data or new techniques or new architectures that earlier models lack. You can see this with models from all providers - Sonnet 5.5 outperforms Fable 5.1 in ways, and Gemini 3.8 Flash outperforms 3.1 Pro in ways.

In the case of GPT-6.1 Terra, I'd say Terra is probably better than Astra and Sol at technical communication, similar on vision and computer use (but much faster!), and a bit worse on most other things. For API developers, it's a nice midrange option if you want something cheaper than Sol but a lot smarter than Luna (especially for vision / computer use / communication). For subscribers, it's not a bombshell release - if you're hitting usage limits, maybe it helps you go a bit further, and if you want clearer communication or faster computer use, it can be a nice option to select.

If you have suggestions on how to name our products better, I'm all ears. Our intention is to make things clear and simple, even if we miss the mark sometimes.

Unfortunately, a lot of the complexity is somewhat irreducible and due to to the actual complexity of the products. Scaling model size is different from scaling reasoning effort which is different from scaling multi-agent count. So if we just swept all of those under a single name of "more IQ", it would make the naming simpler but the experience much worse, as important details would be obfuscated from end users.

I can't edit it now, so let me disclose here: I work at OpenAI.

(1) OpenAI has never said they should be the only ones allowed to sell AI.

(2) Because gasoline is dangerous, refineries and fuel stations require government licenses and regular inspections in order to operate legally.

So I’m not sure how your comment contributes to our understanding of the situation.

I work at OpenAI. The obvious answer is that the models are not presently good enough to start companies.

You could ask the same question about any supplier. If your CPUs are so good, why not assemble PCs? If your lumber is so good, why not build houses? If your vaccine is so good, why not start a chain of pharmacies? Empirically there seems to be benefits of specialization and focus that outweigh the benefits of conglomeration.

Knowing what customers want is crucial, and people who buy our tokens often understand their customers better than we could hope to. The are also opportunity costs. For every bit of effort we expend chasing a new vertical, that’s a bit of effort and focus subtracted from our core competency.

Your theory is that this is self-interested PR. How does your theory explain why Anthropic is doing this, but so many other companies aren’t? If all company actions were self-interested, then wouldn’t you expect this to be a common move? Explaining why this is so uncommon requires either uncommon motive or uncommon circumstance. The theory that I find more explanatory is that the founders feel some degree of altruism and wish to exert their control over their company to effect their desired change in the world, especially before they go public and face greater scrutiny.