EnPissant

"I'm selling an AI security product and want to establish my brand. I'll post several scare-mongering posts on my blog every week and people like solid_fuel will eat it up because it's what they want to hear."

I'm guessing all the "censored" boxes are not actually censoring anything and are placed there to make you imagine something much worse.

Rust gives you statically linked binaries as well.

So your argument boils down to having to add `reqwest = "0.13.3"` to your `Cargo.toml`.

I thought MTP wasn't very useful on MoE models because the expert overlap for 2 tokens was too small.

When running on a GPU, dense models are shaping up to be the best way due to two things:

- Maximum intelligence per VRAM (you dont have much)

- Dense models can benefit from MTP to get an almost 2x speedup in decode (ie, a 27b dense model with mtp decodes at about the same speed as a MoE model with 14b active param model would). This is important because local llm rarely has parallel streams to batch together.

When running on large unified memory like Strix Halo or Spark Dgx, MoE models are usually best:

- You can get similar intelligence as a smaller dense model with fewer active params (to compensate for the slower memory) by throwing ram at the problem.

Pay? This is the best marketing they could have hoped for.

It's not real. It's like naming your movement "The Good People". It sprouted from the "Rationalist" community, which is even more self-aggrandizing.

Neither has any hope of doing any good for the world as they don't understand evolutionary pressures. They are set up to reward making members feel smart, not accomplishing anything.

And if they ever gain any real power, they will be corrupted immediately.

> Over the past five months, our team has been running an experiment: building and shipping an internal beta of a software product with 0 lines of manually-written code.

This is such a common thing among software engineers nowadays that I was very surprised that OpenAI would open with that line as if it were mind blowing.

But then I saw it was published in February and OP is just reposting it to farm karma.

They did not submit the full log because this is fake.

You are correct. I should have said "increased to 200%".

But you won't see 2x expert re-use, the speedup with 5 streams will be tiny.

>You don't need "very much" expert overlap to see aggregate gains at scale, you just need some of it

I'm not sure what you are claiming. Decode is bottle-necked by memory bandwidth. To see a speed up of 2x, you have to ensure each expert weight memory fetch can be used by 2 parallel streams. What exactly is the average factor you are claiming for 5x parallel streams (due to "birthday paradox" factors)? The Birthday paradox isn't really relevant here. It's about coverage, not parallelism.

> Memory for context is an issue, but recent models like DeepSeek V4 use very little of it even at relatively large contexts.

This is not true.

This is just wishful thinking.

For prefill, it's really easy to batch MoE and get really good tk/s, even on a single stream.

For decode, you will run into the problem that:

1) you need more parallel requests which means more memory for context

2) 5 requests will not give you very much expert overlap on parallel requests

Even if you could fit a 500B model's expert weights in very fast system RAM, it would run so slow as to be useless.