Comment by domenicrosati

Cool! What motivated you to write it? I find the d2l implementation nice and simple as well https://d2l.ai/chapter_attention-mechanisms/transformer.html

The one in pytorch itself isnt so bad either.

Be nice to see some examples of decoding in your repo (forgive me if I dont see it). I remeber when i first implemented a transformer from scratch generating sequences using greedy or beam search after train/testing turned out to be harder then I thought but turns out I made a mistake with teacher forcing in the beginning so BOS tokens were meaningless to the decoder lol