Comment by hellohello2

I though 4 bit (mixed) quant is pretty standard, at least for local models that basically what everyone runs? Unless I'm mistaken it can really be a negligible difference in perf.