popinman322

The model license requires large inference providers to contact Meta separately to obtain a license to serve the model. I know we've seen this before with other large models, but it's still a bit irritating.

The usual effect of separate licensing is decreased price competition from public providers (like we saw with Kimi K3). I guess it's better for the providers if the price is fixed: any efficiency improvements directly improve margin rather than driving prices down over time. Still, to provide my own anecdata, the main thing keeping me on a paid subscription is token costs for the more capable models.

These kinds of alignment problems remind me of times where someone does something that's trivial for them but very hard for the recipient. They might say something like "this must have taken you days" when the task really took 15 minutes.

What's the difference between an API search and a DNS workaround from the model's perspective? I think for most humans the DNS workaround is discarded because it's obviously too much work, not because it's untenable. With the vast knowledge base in the latest models, the cost difference falls sharply; it knows what to do and can do it for a very reasonable cost to itself.

General alignment seems to typically focus on high level value questions. Here, we're dealing with an effort alignment issue where values diverge because the solution effort is different for models vs humans.

With RAM prices how they are right now the used laptops might be more valuable than $1500.

Very much agree. Until the vibe-coded version has been fully audited and profiled to perform, within reasonable tolerances, as well as the original code base, it feels like a bad idea to support it downstream or use it in production.

Oh, this is great!

I've filed bugs with JetBrains before and had them take months getting to my ticket, often with multiple hand-offs between team members; being able to provide a potential fix should make the process much faster.

Does anyone know whether we'll be receiving transcoders for this batch of models? We got them for Gemma 3, but maybe that was a one-off.

I've found that Gemini models often produce pseudocode that seems good at first glance but is typically wrong or incomplete, especially for larger or more complex functions. It might produce pseudocode for 70% of the function, then silently drop the last 30%. Or it might elide the inside of switch blocks or if statements, only including a comment explaining what should happen.

Alternatively, Claude Opus generally output actual code that included more of the original functionality. Even Qwen3-30B-A3B performs better than Gemini, in my experience.

It's honestly really frustrating. The huge context size available with Gemini makes the model family seem like a boon for this task; PCode is very verbose, impinging on the headroom needed for the model's response.

It doesn't look like the code anonymizes usernames when sending the thread for grading. This likely induces bias in the grades based on past/current prevailing opinions of certain users. It would be interesting to see the whole thing done again but this time randomly re-assigning usernames, to assess bias, and also with procedurally generated pseudonyms, to see whether the bias can be removed that way.

I'd expect de-biasing would deflate grades for well known users.

It might also be interesting to use a search-grounded model that provides citations for its grading claims. Gemini models have access to this via their API, for example.

They're comparing against open weights models that are roughly a month away from the frontier. Likely there's an implicit open-weights political stance here.

There are also plenty of reasons not to use proprietary US models for comparison: The major US models haven't been living up to their benchmarks; their releases rarely include training & architectural details; they're not terribly cost effective; they often fail to compare with non-US models; and the performance delta between model releases has plateaued.

A decent number of users in r/LocalLlama have reported that they've switched back from Opus 4.5 to Sonnet 4.5 because Opus' real world performance was worse. From my vantage point it seems like trust in OpenAI, Anthropic, and Google is waning and this lack of comparison is another symptom.

You can swap experts in and out of VRAM, it just increases inference time substantially.

Depending on the routing function you can figure out all the active experts ahead of the forward pass for a single token and pipeline the expert loading.

The executive branch is currently ignoring the law. Why would they start following it in 2029?

Not a fan of censorship here, but Chinese models are (subjectively) less propagandized than US models. If you ask US models about China, for instance, they'll tend towards the antagonistic perspective favored by US media. Chinese models typically seem to take a more moderate, considered tone when discussing similar subjects. US models also suffer from safety-based censorship, especially blatant when "safety" involves protection of corporate resources (eg. not helping the user to download YouTube videos).

Assuming you're doing local inference, have you tried setting a token filter on the model?

DeepSeek was built on the foundations of public research, a major part of which is the Llama family of models. Prior to Llama open weights LLMs were considerably less performant; without Llama we might not have gotten Mistral, Qwen, or DeepSeek. This isn't meant to diminish DeepSeek's contributions, however: they've been doing great work on mixture of experts models and really pushing the community forward on that front. And, obviously, they've achieved incredible performance.

Llama models are also still best in class for specific tasks that require local data processing. They also maintain positions in the top 25 of the lmarena leaderboard (for what that's worth these days with suspected gaming of the platform), which places them in competition with some of the best models in the world.

But, going back to my first point, Llama set the stage for almost all open weights models after. They spent millions on training runs whose artifacts will never see the light of day, testing theories that are too expensive for smaller players to contemplate exploring.

Pegging Llama as mediocre, or a waste of money (as implied elsewhere), feels incredibly myopic.