Comment by rdedev

Doesn't using bf16 alleviate the problem? At least I've had success training a Bert like model from scratch