Meanwhile, entirely too frequently I see unit tests that have atol / rtol values which permit errors of one part in 200 or even higher, where people bother with unit tests at all. At some point someone in the pytorch ecosystem used rtol=5e-03 in a unit test and crowds of people started passing that around unquestioned.
With attention matrix sizes being what they are, that's high enough that you can literally zero out a row or two and still have the test suite pass. Guess how I know!
With attention matrix sizes being what they are, that's high enough that you can literally zero out a row or two and still have the test suite pass. Guess how I know!
Check your numbers, folks.