Models are not like a fps counter where losing a few percentage has no actual impact. These percentages may be the difference between a model that writes code and one that goes into loops, and think rm -rf . is a good idea.
There is a reason why most models try to stay in the FP4 or higher range, because the reduced accuracy can have major consequences.
You are better off with a 8b FP4+ model then a 27b Q2 model.
There is a reason why most models try to stay in the FP4 or higher range, because the reduced accuracy can have major consequences.
You are better off with a 8b FP4+ model then a 27b Q2 model.