Comment Comment by vladf Thanks for the reply. I’m quite familiar with subchannel quant, but still feel like my questions did not get addressed.1 Could you post the full memory use of the methods? E.g. you include quip metadata in its GB but not hqq metadata in its GB.2 If you have to go to cpu to shift and scale, how did you get latency lower than pure on device? Was this bsz1? No speculative decoding?3 how can lora absorb shifts with only increasing rank by 1 if you have a shift per group? parent vladf 2 years ago
1 Could you post the full memory use of the methods? E.g. you include quip metadata in its GB but not hqq metadata in its GB.
2 If you have to go to cpu to shift and scale, how did you get latency lower than pure on device? Was this bsz1? No speculative decoding?
3 how can lora absorb shifts with only increasing rank by 1 if you have a shift per group?