Comment by omneity

Attention is calculated during the forward pass of the model, which happens in both inference (forward only) and training (forward & backward).