upbeat_general

And if there are differences, that is often a good thing! It can mean the author wanted to format something in a particular way that the journal didn't allow.

My uneducated guess is that it’s done to avoid clutter for the rest of the settings and because accessibility is a good justification to outweigh this concern.

This is just not how it works - it is perfectly plausible (in fact the most likely) that pretraining was well finished by the time Fable was released. A bit of extra distillation takes far less compute.

Moreover, it’s very plausible (and expected) to use multiple clusters and GPU types for RL rollouts which could very well not be included in this count.

No part of this pipeline is fixed in stone.

Same way you do for planes, cars, etc. You have long(ish) life-cycles and some pieces that can be independent (e.g., same turbine engine on several plane models).

If I used a racecar to go 25mph in a residential neighborhood, I’d make a similar conclusion.

I think that figure (haven’t verified it but assuming it’s true) isn’t complete. It hides who and where those people are - for example, I imagine art history skews towards higher ranked schools in the first place.

If we’re defining on-prem as fitting in a rack - then every frontier model can be hosted on-prem.

Now this might not be the most cost effective (and may require a bit extra power), but you only need a datacenter for training or cost optimization.

This statement is well known to be incorrect for at least a year.

If by "happened to" you mean pour significant resources for well-over a decade on many different AI research groups then yes, that's accurate. Depending on your definition of AI, it might even be two decades.

In fact, OpenAI was founded largely with the direct goal of preventing Google from being the sole winner in AI...

This isn’t quite accurate. Data weighting is quite important in pretraining.

Sampling exists.

From a user perspective it is nice to know that the person decoding will likely support a given format, both now and in the future.

More use cases for a single popular format makes this more likely.

I really wish that there was an “almost yolo” mode that was permissive but with light restrictions (eg no rm), or even better, a light supervisor model to prevent very dangerous commands but allow everything else.

I can assure you that lacking knowledge in DPO (and especially GRPO it’s just stripped down PPO) is not a dealbreaker.

This seems like the perfect case for legislation that starts out targeting higher volume devices/larger companies and lower over time.

I don’t see why the industry couldn’t move to providing this documentation/full source over a few years.