I struggle with approaching this as I do with many quants, in that there's not really a good measurement of how lobotomized by quantization they are other than "a bit but it's not too bad". Ternary weights immediately make me think "unusably quantized" but I'm presuming that's not the case if someone's standing by this release without massive caveats.
I struggle with deciding whether to take the speed hit on q8 vs q6; I wish it was more standard to benchmark quants or have some way to indicate how much intelligence you're leaving on the table.
I don't mind waiting twice as long for twice as smart; I don't want to be passing up the opportunity of 4x speed-up with 98% intelligence retained. But I have no way to differentiate which is which with quants in any vaguely objective way.
I struggle with deciding whether to take the speed hit on q8 vs q6; I wish it was more standard to benchmark quants or have some way to indicate how much intelligence you're leaving on the table.
I don't mind waiting twice as long for twice as smart; I don't want to be passing up the opportunity of 4x speed-up with 98% intelligence retained. But I have no way to differentiate which is which with quants in any vaguely objective way.
Frustrating, as a hobbyist.