amanrs
Joined 6 years ago
Karma
3
Submissions
5
Member since
Apr 12
Recent submissions
Comment · 1 year ago
Comment · 2 years ago
It edits the file based on the "plan" laid out by a smarter language model
Comment · 3 years ago
The key misconception about many quantization methods is that lower precision = better speed.
I believe GPT-Q is not much faster than bf16 from skimming the AWQ paper - https://arxiv.org/pdf/2306.00978.pdf
It's 3x faster for a batch size of 1, but that's still over 10x more expensive than gpt-3.5
For larger batch sizes, bf16 costs dip below 3-bit quantized.
First "token-healing" doesn't work. Consider the case "app" where the most likely options are "ap|praisal" or "apple|sauce". You can't just sample all tokens that start with app, or you'd miss appraisal.
Second, it's easy to come up with a naive algorithm that samples from the true distribution. It's very difficult to make this algorithm efficient.