My current mental model of LLMs is that attention refines information from the input (e.g. "red balloon" becomes some vector that means red + balloon), and that the feed-forward layers are lossy compressed indexes into the training set.
From the paper, having a better n-gram index is giving as much perplexity improvement as a 10x increase in parameters. I wonder if it would make sense to train new foundation models with more attention heads and use infinigram in lieu of the feed-forward layer.
From the paper, having a better n-gram index is giving as much perplexity improvement as a 10x increase in parameters. I wonder if it would make sense to train new foundation models with more attention heads and use infinigram in lieu of the feed-forward layer.