Comment by SlinkyOnStairs

Realistically, we are.

This is not some arbitrary design choice, it's the core compromise to make LLMs viable to train at all.