i created something similar a while back. Inspired by micrograd for showing the connection between math/calculus and code, then building along the way to a full NumPy deep learning library that I pretrained GPT-2 124M model with it. One way to learn is to trace through the PRs merged to the repo in chronological order.
The philosophy is the same: "What I cannot create, I do not understand." https://github.com/workofart/ml-by-hand