ianbutler

It's not a weighted average though. I see this repeated all the time. Between synthetic data, custom produced training data by experts, post-training regimes etc the models are far more diverged from that baseline at this point, at this point it's mostly a right shifted curve.

Idk, your limitations section sure makes it seem less drop in and less general than Jev. Like the point here isn't your ML aptitude it's how easy is it for developers to drop this into a product and use it.

I'm more than capable of training a bert classifier in fact in 2019 I had trained many custom berts and was running them on hundreds of millions of documents a day.

I don't want to manage GPUs / CPUs now. I don't want to maintain my corpus and retrain as my product's data distribution shifts. The list of things I don't want to do goes on and on and on. And I'm happy for them to be someone else's problem.

I do just want a reasonably good general classifier served to me with a great devex and calibrated confidence scores to help me figure out when to fallback to another model.

I see this super interestingly as the "subconscious" to the llms "conscious" for lack of better terms. I'm super interested in this for broad and rapid decision making in the context of consumer agents so will be signing up for sure.

> Very recently, I've started avoiding produce from Taylor Farms Good thing I was paying attention to the news when that mess started.

This is very obviously not the type of situation and behavior marginalia_nu and then I were commenting about.

> Reality sucks and sometimes you have to do something about it or else you're the problem.

But ARE you doing anything about it? Because unless you're actually taking action it's basically just self flagellation to the benefit of no one. Like if you have some measure of power (community action, organized resistance, political lobbying) and are taking action that's great, reading articles that make you sad and angry and then going to bed isn't doing anything.

What are you doing to fix the issues though? Has the knowledge of the issues increased your agency in a meaningful way such that you've taken actions to improve things or are you just enjoying living in a perpetually outraged state for the sake of it? If your martyrdom isn't doing anything to help fix the problem you're just hurting yourself and stopping yourself from potentially being in a space to ever actually do anything.

I see researchers, tech people and my friends. This is a problem with how you curate your social media.

I don't think this is really entirely true. Everyone who isn't a FAANG or Frontier Lab seem to be absolutely starving for good talent. It's far more often that I hear "we interviewed someone we thought looked good and they couldn't even write a for loop, where are all the good engineers".

I think you have far more leverage than you think in most places still.

> But long-context is a trap, because performance still falls dramatically after 150k-200k context.

I often see this repeated, and it is not true task to task. I work on this daily and we have several tasks where long context is advantageous and our evals against a whole battery of models with different windows show it as being so.

This is why having good evals for the tasks you're working on is so important.

I do grant it's a good rule of thumb.

It's nice to see a strong long context open weights model that is multi-modal.

There are many applications that will benefit from the strength in audio here and until z.ai and co work in visual this could be very strong for general agentic applications, though I see there's a bit of weakness in the benches for areas that might make that less true.

Like all models need to slap it in your harness and do proper evals on the tasks you care about.

There's a setting that stops fallback to Opus 4.8 if you would like to avoid that behavior.

Config -> Switch models when a message is flagged -> false

That should stop it entirely when a message is flagged and then you can come back without Opus having potentially made a mess.

Most of the best engineers I know leave faang because the work at faang sucks. It is not orthogonal. The majority of faang work is not highly impactful work like trivially because the amount of work outstrips the amount of interesting work by an easy oom.

FAANG doesn't straightforwardly retain the best technical talent, and the median task there is routine.