Comment by sigmoid10

Earlier this year there was a paper from Microsoft where they trained a 1.58 bit (every parameter being ternary) LLM that matched the performance of 16 bit models. There's also other research that you can prune up to 50% of layers with minimal loss of performance. Our current training methods are just incredibly crude and we will probably look back on those in the future and wonder how this ever worked at all.