Comment by jstx1

Yep, for so many NLP and vision tasks it makes sense to use a pretrained model as a preprocessing step and train a couple of dense layers with your own dataset on top of it.