MDK8888
Joined 2 years ago
Karma
42
Submissions
8
Member since
Jan 5
Recent submissions
Comment · 2 years ago
I built vllm from the ground up. Starting from the same kernels, I integrate them into GPT2, build the KV cache manager, the scheduler, and finally the FastAPI server on top. This is purely educational for now, but in the future it will become production ready.
Comment · 2 years ago
This has been added to the Readme under the 'Background' section.
Comment · 2 years ago
Hey, hope you're doing well! I've updated the repo with instructions on how to get started and documentation!
Comment · 2 years ago
A few months back, the PyTorch team released GPT, Fast (https://news.ycombinator.com/item?id=38477197), a collection of techniques to 10x the inference speed of Llama-2-7b.
This library generalizes those techniques (quantization, torch.compile, speculative decoding) to all Huggingface models, achieving an inference speed speedup of 6-7x.
As more optimization algorithms are discovered, I will post updates to the repo to HN.
Comment · 2 years ago
SageMode is an open source python library for deploying, scaling, and monitoring your LLMs at scale. It's designed to be highly flexible but also standardized. You can deploy any Huggingface Model that exists in version 4.26 or earlier as well as any PyTorch Model. If you find any issues with it, open an issue-I'd love to iron out any problems.