Comment by treyd

The unreasonable effectiveness of transformer attention outweighs a lot of the downsides of limited context length for many applications.