Comment by kromem

While it's true that the feed forward is one word at a time, the self-attention is not, which is the key difference transformers brought to the table.

It's kind of kind saying "computers can add." Yes, they can. But they can multiply too.

Over the past 18 months transformers have proven to upend a number of longstanding assumptions, particularly around world modeling.

It's a very misleading statement to describe today's LLMs as "simply predicting the next word" even if it's in part true.