Comment by florianletsch

Sidenote: A common practice is to take a pre-trained model (e.g. on the Imagenet dataset) and only learn the last few layers for your usecase. This way you can get a well trained feature extractor if your task data is similar, and then only train the classification, which is a lot faster than full end-to-end training.