Comment by harisec

Yes, during training multiple checkpoints are created, you can distill from any checkpoint you want.