lcampbell

OG DeepSeek V4 Flash and V4.1 Flash are actually completely different architectures. To wit, V4 runs fine on 2x96GB cards but you’re not running V4.1 unquanted on less than 4x96GB.

It’s not really an apples-to-apples comparison.

(I’m not sure why they didn’t call it V5 Flash, but model names from most labs are completely nonsensical and disjoint from their underlying architecture.)

I was under the impression that the cache write fee was added to both the input and output costs (except in cases where the cache write is explicitly disabled via e.g. DISABLE_PROMPT_CACHING). The output becomes part of the context, after all; if they don't (for some reason, due to disaggregated inference perhaps) then I'd expect output tokens get charged both output then input+cache_write on the subsequent completion request.

The pricing model confuses me though (I presume by design, Hanlon be damned).

> Do all other providers somehow overcharge by that much?

This, I think. Cached inputs have an opportunity cost (keeping the KV cache until use) but a hit is basically free. “Basically” - if the cache is offloaded to system RAM or NVMe there’s some scheduling overhead.

From a consumer viewpoint a more interesting metric than the raw costs is

    cached cost * hitrate + input cost * (1 - hitrate)

from a personal standpoint rather than a per-provider one (e.g. if OpenRouter is blindly dispatching your requests you might have a bad time).

> Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users)

Is that actually the aggregate throughput? 8xB300 (with 4TB/s/GPU bandwidth) is only pushing 8 tg/s/session? That seems… incredibly low, even for an A100B model.

Is it actually 122 tg/s per session? (1952 tg/s aggregate throughout)?

More than that, the AI disclaimer itself is a hallucination:

> Gemini 4 helped with the editing.

There is no Gemini 4, unless the author is writing from the future.

FWIW, you can get a 16x RTX 6000 Pro setup running for significantly less than that. Even considering the electrical hookup fees (it’s a lot of power and cooling). Home ownership might push you into the original quote though.

> The fix that worked is a schema transform at the provider boundary. For OpenAI-family models only, we rewrite every optional property to be required but nullable, using anyOf: [T, null], which gives the model an explicit way to say “not using this.”

I admit, I've only used a bastardized form of MCP, but this smells... wrong? It's not clear to me why the Typescript type definitions would have any influence on (what I presume is) JSONSchema being sent from the agent to the inference backend as part of the completion request. The MCP specification (which the OpenAI backend might not use, I don't know) has an explicit field to signify "optional" parameters in the JSONSchema; my read on this is there's a bug somewhere between the Typescript layer(??) and the generated tool description which is actually sent to the inference backend.

It's possible the inference backend has changed from "generate valid tool responses" to "generate valid tool responses according to the JSON schema [where no parameters are optional]" but it's impossible to tell without seeing the actual requests sent to the inference backend (which I didn't see in TFA).

> I want to be careful here.

was the giveaway for me

The reasoning generally isn't kept in the context, so after choosing the secret word in the first reasoning block, the LLM will have completely forgotten it in the second and subsequent requests.

So, it technically didn't change the secret word so much as it was trying to infer what its own secret word might have been, based on your guesses.