Comment by tarruda

You can fine tune without the pre training data too.

Mistral models are one example, they never released pre training data and there are many fine tunes.