Comment Comment by rewq4321 > Also the attention mechanism is baked in during pretrainingIIUC, this is no longer necessarily true with positional encodings like ALiBi: https://github.com/ofirpress/attention_with_linear_biases parent rewq4321 3 years ago
IIUC, this is no longer necessarily true with positional encodings like ALiBi: https://github.com/ofirpress/attention_with_linear_biases