Comment by snorkel

Strongly recommend watching Andrej Karpathy’s “Lets build GPT-2” videos on YouTube which dives into an actual PyTorch implementation, then download the code and study it carefully. Then study “Spreadsheets is all you need” to see what the internal data structures look like.