Comment by ironrabbit

Andrej's "Building Makemore" series is exactly this, it includes a wonderful lecture where he computes all the gradients for a simple network by hand and compares them against the values produced by torch's autograd.