abcdabcd987
Joined 11 years ago
Karma
122
Submissions
29
Member since
Aug 14
Recent submissions
Comment · 2 years ago
Related discussion on serving finetuned LLMs: https://news.ycombinator.com/item?id=38196661
Comment · 3 years ago
Thanks for your encouragement! We are working on quantization as well. We recently submitted a paper, Atom [1], that uses 4-bit quantization, delivering 7.73x throughput compared to FP16 and 2.53x compared to INT8. Atom is able to maintain a perplexity (i.e., model accuracy) close to FP16, outperforming existing quantization approaches.
We are polishing the 4-bit code. It will be added to Punica code base soon. Please stay tuned :)
Comment · 3 years ago
Certainly! We'd like our good designs to be picked up by frameworks and serve all users. Currently, Punica is built on top of PyTorch and HuggingFace Transformers ecosystems. Therefore, vLLM and LMDeploy, which are also in the PyTorch ecosystem, should have a smooth adaption. As for Nvidia Triton and TensorRT-LLM, since our kernels are written in CUDA, I believe it will also work seamlessly.
We call for the open source community to help us integrate Punica with all frameworks, thus the whole society can benefit from the efficiency improvement!
Comment · 3 years ago
Thank you! We are also very excited about combining the fast fine-tuning and efficient serving. In fact, what you just said is very related to one of our very first motivations. In my previous blog post [1], I call this scheme "Just-in-time Fine-tuning". Our previous measurement is that, for a medium-sized webpage (~10K tokens), it takes around 30 seconds to 2 minutes to finetune a LoRA model. Another good side of this JIT fine-tuning scheme is that, we can turn any model into a long-context model.
We'll keep doing more research on finetuning. And hopefully, we'll see the results soon.
[1] https://le.qun.ch/en/blog/2023/09/11/multi-lora-potentials/
Comment · 3 years ago
Thanks for the question. Currently Punica is on the ecosystem of PyTorch and HuggingFace Transformers. So PyTorch users can start to use Punica now.
Look forward to collaboration with TVM and MLC to reach more users :)
Comment · 3 years ago
On a related note, Google also uses optical circuit switches in their datacenter network. See the paper form SIGCOMM'22 [1].
[1] Jupiter Evolving: Transforming Google’s Datacenter Network via Optical Circuit Switches and Software-Defined Networking. https://research.google/pubs/pub51587/
Comment · 3 years ago
I'm using 27-inch 4K monitors with Gnome. I found it more practical to use no scaling but 1.25x text size (settings in gnome-tweaks). The problem for me with 2x scaling or fractional scaling is that it scales UI (icons, margins) as well. As someone who appreciates functionality (i.e., displaying text) over overdesigned white spacing, scaling the whole UI is just wasting my workspace size whereas scaling text-only is a perfect balance between good-looking text and workspace size.
But anyway, good to see that Linux desktop is gaining fractional scaling support!
Comment · 5 years ago
I use Automate[1] to do this. Just draw an infinite loop that does sms_received[2] and email_send[3].
[1] https://play.google.com/store/apps/details?id=com.llamalab.a...
[2] https://llamalab.com/automate/doc/block/sms_received.html
Comment · 5 years ago
> If you find a really good tutorial by programmers who are both excellent teachers and experienced in that particular field, it beats JIT learning on a personal project in one important respect. You get exposure to the One True Way of doing things.
That's exactly my feeling when I was watching Jon Gjengset's Rust tutorials. I like his real reactions to unexpected problems. Really learned a lot from this kind of lengthy but realistic videos. https://www.youtube.com/c/JonGjengset/featured