ani17
Joined 1 year ago
Karma
9
Submissions
16
Member since
Sep 22
Software Developer at a computer networking company. Interested in Distributed Systems, Networking and ML.
View on Hacker NewsRecent submissions
Comment · 6 months ago
Author here. I wanted to understand what vLLM and llama.cpp are actually doing under the hood, but the codebases are massive. So I wrote a stripped down version from scratch to see the core ideas without the production complexity.
Comment · 7 months ago
The blog walks through why your first token is always the slowest, why output tokens cost 5x more, and how stuff like speculative decoding and chunked prefill actually work, from the perspective of a systems engineer!
Comment · 7 months ago
Author here. A bit more context: By day I'm a systems engineer building AI networking infrastructure. So I kept ending up in conversations where I'm not exactly able to wrap my head on the latest inference magic trick.
Like when someone mentioned vLLM's paged attention, I knew virtual memory paging, but had no idea someone had applied the same idea to KV cache allocation on GPUs.
Github link to the project: https://github.com/Anirudh171202/WhiteLotus
Comment · 11 months ago
Definitely an alternative solution. For the purpose of this script, I wouldn't prefer that though.
Comment · 11 months ago
It's insane if the data is accurate. Only time will tell
Comment · 1 year ago
You forgot "Middle Out" by Pied Piper!
Comment · 1 year ago
thanks for sharing!