Elon says Opus is 5T (and I would expect he'd know)
> It's not that frontier labs can't create a 5T+ parameter model, but they don't have the data to optimize a model of that size.
The have plenty if data. They use very large amounts of verifiable synthetic data in (lots in coding and math) cover the gap.
Also the frontier labs are paying people to do tasks, tracking the trajectories and training on that. Most of the optimization is in RL based on these trajectories.
> It's not that frontier labs can't create a 5T+ parameter model, but they don't have the data to optimize a model of that size.
The have plenty if data. They use very large amounts of verifiable synthetic data in (lots in coding and math) cover the gap.
Also the frontier labs are paying people to do tasks, tracking the trajectories and training on that. Most of the optimization is in RL based on these trajectories.