Comment by NooneAtAll5

I wonder if instead of outright masking bad tokens, you could've done the training while slowly increasing negative weights on those