Also the attention mechanism is baked in during pretraining. So whatever max context length you want increases the compute cost of training by at least a function of said "bad complexity." Even just 4096 tokens of max context is much more expensive to train than 2048. So if we want models with 8k, 32k, or more context then the training costs get out of hand quickly.
Because of exactly that.
Also the attention mechanism is baked in during pretraining. So whatever max context length you want increases the compute cost of training by at least a function of said "bad complexity." Even just 4096 tokens of max context is much more expensive to train than 2048. So if we want models with 8k, 32k, or more context then the training costs get out of hand quickly.