Comment by sigmoid10

Attention just allows the model to attend to elements across the input sequence. The feed forward is what gives it (and basically all other architectures) its universal function abilities. The only reason why we don't use dense layers directly across the input sequence and instead go for things like convolution, recursion or transformers, is because that is prohibitively expensive computationally. But replacing the dense layer with n-grams would make LLMs exactly what so many people falsely believe them to be right now: Pure stochastic parrots instead of functions that can actually learn to generalize from examples.