Comment by om8

> If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.

That’s why you need to use efficient gemm kernels like FLUTE for inference. They are ~as good as what you can do with ternary quantization.