Comment by sebzim4500

Don't transformers typically have a <bot> token at the beginning of the prompt? This seems equivalent to letting the network attend to this token, and produce a zero value if that's what it wants.