If you could train models faster, you’d be able to build larger, more powerful models that outperform the competition.
The fact that Llama 3 is significantly over trained than what was considered ideal even three years ago shows there's a strong appetite for efficient training. The lack of progress isn’t due to a lack of effort. No one has managed to do this yet because no one has figured out how.
I built 1-trit quantized models as a side project nearly a decade ago. Back then, no one cared because models weren’t yet using all available memory, and on devices where memory was fully utilized, compute power was the limiting factor. I spend much longer trying to figure out how to get 1-trit training to work and I never could. Of all the papers and people in the field I've talked to, no one else has either.
If you could train models faster, you’d be able to build larger, more powerful models that outperform the competition.
The fact that Llama 3 is significantly over trained than what was considered ideal even three years ago shows there's a strong appetite for efficient training. The lack of progress isn’t due to a lack of effort. No one has managed to do this yet because no one has figured out how.
I built 1-trit quantized models as a side project nearly a decade ago. Back then, no one cared because models weren’t yet using all available memory, and on devices where memory was fully utilized, compute power was the limiting factor. I spend much longer trying to figure out how to get 1-trit training to work and I never could. Of all the papers and people in the field I've talked to, no one else has either.