Comment Comment by anushka_gandhi No, you require BF16 model weights for this quantization to work. If you do run it on 4bit or 1.5bit, it will degrade the model and it will hallucinate a lot. parent anushka_gandhi 12 days ago