apejcic
Joined 7 years ago
Karma
21
Submissions
7
Member since
Oct 22
Recent submissions
Comment · 24 days ago
Comment · 1 month ago
Author here. We serve several tenants on shared SGLang nodes, and our p99 TTFT was bad while the median looked fine. The cause was head-of-line blocking in prefill. SGLang's scheduler lets only one request be mid-chunk at a time, and that request takes the whole chunked-prefill budget every round. So one cold 600K-token prompt held the prefill slot for about 105s. Everything queued behind it waited too, including requests with a 99% prefix-cache hit that had almost nothing left to prefill.
The fix turns that single slot into a set, capped per request by long_prefill_token_threshold. Most of the work was in the places that assumed only one partial prefill could exist: KV reservation, output streaming, parking chunks under memory pressure, and spec-decode tail tokens.
In that scenario the big request slowed from 105s to 112s. The six requests behind it went from about 105s to between 0.8s and 2.4s.
It isn't free. If every request is huge, FCFS is already the right policy and chunking only adds overhead. A lone long prefill still pays the per-round cap even when nothing else wants the budget. Peak KV usage also rises somewhat. The threshold is a ceiling today, and I'd rather it were a floor, so concurrency adapts at runtime without tuning.
Upstream PR: https://github.com/sgl-project/sglang/pull/34623
I'm happy to go into scheduler details. I'd especially like to hear from anyone who has handled this differently, for example with prefill/decode disaggregation or priority queues.
Comment · 3 months ago
True
Comment · 3 months ago
You don't have to use Claude Code. Use PI (fully local harness) with ZDR open weight model provider (eg like sference.com).
I'm getting negative reaction for providing an obvious solution. Anthropic stores all your prompts as well and uses it for training.
Comment · 3 months ago
Use GLM-5.2 on ZDR inference provider like sference.com
Comment · 4 months ago
Having led engineering in SF big tech and now building my own startup, I’ve heard this exact argument countless times. I believe it is precisely why large tech companies are losing their ability to innovate.
Engineering shouldn't be an academic pursuit of "good engineering" for its own sake. It is fundamentally a business function. The objective is to achieve real world goals and build things people actually want, even if that means the solution is scrappy and unpolished today, so long as it solves an important problem cheaply and effectively.
In large organizations, accountability for actual value delivery is so diluted across the org chart that it practically disappears.
Leaders gravitate toward building bloated "internal platforms" that offer little external utility. Engineers over-engineer simple problems because their performance reviews reward technical complexity, not business impact. It becomes a systemic shield, allowing the old guard to justify their headcount without delivering new value.
Software engineering is uniquely plagued by this. Civil engineers understand their objective: build a bridge that safely carries traffic within a specific budget. They don't invent a novel, hyper-complex suspension architecture when a standard short-span bridge will do the job.
- Reasoning effort = 1 is now actually usable and produces 2.5x less tokens than old weights, without intelligence lost.
- Overall 20% less tokens at each reasoning effort.
- Same intelligence and same tool-calling pattern.
More details in blog post https://sference.com/blog/deepseek-v41f-1017-benchmark