"Attention with Linear Biases (ALiBi): Inherited from the MPT family, this feature eliminates the context length limits by replacing positional embeddings, allowing for efficient and effective processing of lengthy documents. In future we are planning to finish training on our larger dataset and to increase amount of tokens for context."
This is interesting, but also confusing. What is the current context limit? It mentions eliminating the limit but then mentions increasing it in the future.
This is interesting, but also confusing. What is the current context limit? It mentions eliminating the limit but then mentions increasing it in the future.