FranckDernoncou
Joined 10 years ago
Karma
138
Submissions
8
Member since
Oct 8
http://francky.me
View on Hacker NewsRecent submissions
Comment · 5 months ago
Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author.
Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the base model.
Results:
- Up to 7.8x TPF, ~6x wall-clock on MATH-500.
- 16% of params trained, <1B tokens, 24h on 8xH200.
- vs. diffusion LMs (Dream, Fast-dLLM-v2, SDAR, Mercury, Gemini Diffusion): they modify base weights and lose accuracy (Fast-dLLM-v2: -11 pts on MATH-500). Orthrus freezes the backbone; accuracy matches Qwen3-8B exactly.
- vs. Speculative Decoding (EAGLE-3, DFlash): no external drafter, no separate cache, zero TTFT penalty (no drafter to init/sync). KV overhead is O(1) (~4.5 MiB flat). Acceptance length on MATH-500: 11.7 vs. 7.9 (DFlash) vs. 3.5 (EAGLE-3).
- Single-step denoising beats multi-step (6.35 vs. 3.53 TPF). KL distillation beats CE on acceptance rate.
Limitations: strictly bounded by the frozen base model (inherits its biases, hallucinations, knowledge gaps); Qwen3-only evaluation; greedy + rejection sampling only.
Comment · 10 months ago
But harder questions will typically get no replies and/or get closed for being too niche and/or get downvoted because someone got upset they didn't understand the question. That's my experience.
Comment · 10 months ago
> LLMs are terrible, but stackoverflow is definitely worse.
Exactly. Some stackoverflow users keep complaining about AI slop but they fail to realize that human slop is much more rampant.
Comment · 10 months ago
> In fact, SO has a great opportunity to specialize in answering questions that LLMs cannot answer (bleeding edge technology, complex debugging problems, emerging issues, you name it.)
But they are missing that opportunity because such questions typically get no answer and no votes (or sometimes even downvoted as some users tend to downvote questions they don't understand), which caused them to be automatically deleted later. And some of these questions will be closed e.g. as too niche (complex debugging problems). I've posted over 7000 questions across Stack Exchange and that's my experience.
> quick and confident-sounding answers to whatever problem you throw at them.
Just like an answer on Stack Overflow.
Comment · 10 months ago
The entire Stack Exchange network is collapsing: [The number of questions has decreased 10-fold: where did users go and what can be done to prevent that?](https://meta.stackexchange.com/q/408728/178179)
As for Stack Overflow, the number of questions has decreased 25-fold (!) since 2018. A single person could now answer all new questions if they had nothing else to do.
Comment · 1 year ago
The research study is behind a paywall: https://doi.org/10.1038/s41591-025-03773-w
I'm always amazed that some people still choose to publish in paywalled journals.
Comment · 1 year ago
I posted a similar plot showing the steep decline of SO questions on https://meta.stackoverflow.com/a/432618/395857 two weeks ago and surprise surprise, it got removed by a mod... Can still see on https://web.archive.org/web/20250110231518/https://meta.stac...
Meanwhile on https://meta.stackexchange.com/q/384652/178179, it's pretty clear that the few remaining Stack Exchange users are strongly against using LLMs regardless how good LLMs become. Luddites 2.0