nbutton762

Joined 26 days ago

Karma

Submissions

Member since

Oct 6

Recent submissions

First I just want to say, specifically for Post Training Quantization, I absolutely agree with you. As for why 4 bit is the limit, the ParetoQ paper has some interesting theories: https://arxiv.org/abs/2502.02631

That being said, there's a slight misconception about models at lower than 4bpw. There's no fundamental reason why a transformer with low bit weights would be inherently incapable of doing high dimensional function approximation, but training a model at one precision and then quantizing to a lower precision means the training loss is never calculated based on the quantized state.

There's a huge difference between "I trained a ternary model from scratch to do X" and "I trained a model at fp16 to do X and then squashed the hell out of it". Quantization Aware Training is the solve, but it's really expensive compared to a one-time, offline translation of existing weights.

Comment · 23 days ago

Thought this was going to be on the original Little Bit paper, always nice to find out about a surprise sequel!

Comment · 26 days ago

One aspect of how current training and continual learning are somewhat at odds is that the memory required to train a model is often times 2-3x the memory required to just run it (probably not as bad for PEFT, not sure).

DUST does have an advantage specifically along those lines because it doesn't have to save a ton of intermediate state other than each layer's input activations during a single forward pass.

There are many other issues that this algorithm does not address thoigh like catastrophic forgetting. it's still operating on a transformer which contains no inherent mechanism for selecting the relative value of a training step based on current knowledge, nor does it have segmentation of functionalities with specialized areas used for specific things that can be sequestered off and ignore new updates (we do not risk forgetting how to walk as we increase our French vocabulary)