olliepro

Moving L/R is OP

Having crossed paths with developers who have to exert an extreme amount of control over their codebases (such that they rewrite any code they inherit) and having that urgency myself sometimes, I get the premise of LLMs not measuring up, but you can totally care about quality and use LLMs… after all nothing is ever good enough when you need total control.

Even if LLMs aren’t currently up to snuff, I don’t see any deceleration yet in the s-curve. Speed, intelligence, and persistence are all improving every month. I agree that end-to-end prompting “e.g. build me app X” is not production worthy, but i think it’s pretty bold to say coding isn’t solved. AI can code almost anything you adequately specify. The specification is the engineering (it’s a process, not one prompt).

Classic example of moving the goal posts. “Exploiting a loophole” is how you solve many great problems in math.

The authors have some inconsistencies with training token length…

Most errors are probably responses that didn’t finish before their 3K token limit. They’ve measured how well RL is able to shorten the response to their limit.

This is the classic pattern of LLM generated MCQs.

With super high res onboard camera footage too.

They do quite a lot of distillation. As we've seen from the American open weight models from AI2 (OLMo series of models). They have a lot of incentive to distill beyond just copying, they're much more compute constrained, so open model companies distill, but also do really good architectural work to make their models run faster. Theres also technical challenges to distillation when all of the top models have their reasoning traces hidden, so we have to assume these open weight labs also have really great training pipelines as well.

A lot of distillation happens. E.g. OLMo models have a completely open dataset and they are heavily distilled. It only makes sense to try to absorb behaviors from the best models out there. That said, I think the open weight juggernaughts are doing really genuinely great work with RL, training environments, architectural innovations etc.

decentralized training makes a lot more sense when the required hardware isn't a $40K GPU...

This would likely only get used for small finetuning jobs. It’s too slow for the scale of pretraining.

I bet they lack good long context training data and need to start a flywheel of collecting it via their api (from willing customers)

Tensors are in no shortage nowadays. I did read this a tensors though and got a good laugh.

There’s a section of I-15 in Utah’s Salt Lake County which reliably has a crash on weekdays at 6pm. It was unfortunately at a pinch point in the mountains with no good alternate route… very annoying.

In a similar way that Google Maps shows eco routes, it’d be fun for them to show “safest” routes which avoid areas with common crashes. (Not always possible, but valuable knowledge when it is.)

It depends on your thing. If the marathon was just the motivation, your thing is running... if the marathon was the bucketlist item, it is the thing.