Comment by FranciscoCarlos

Still a crutch around incomplete pre-training data, a ton of useful knowledge wont be properly represented on the weights.