Comment Comment by aarnphm Currently on main, 8bit and 4bit quant is supportedOne can simply do```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4```Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization parent aarnphm 3 years ago
One can simply do
```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4```
Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization