It's mostly a scale thing. Transformers scale better when you have a lot of training data. (Not saying CNNs don't scale, but it's way more difficult to tune the hyper parameters).
For smaller training data sizes, transformers don't have a clear advantage over CNNs. CNNs are still the way to go if you have problems that are not too complex, and especially constraints regarding inference costs. Think of a small edge device / sensor that counts apples in your apple picking machine.
What we sometimes see is that a backbone is trained using transformers, then a smaller CNN is destilled from that (DinoV3 for example).
For smaller training data sizes, transformers don't have a clear advantage over CNNs. CNNs are still the way to go if you have problems that are not too complex, and especially constraints regarding inference costs. Think of a small edge device / sensor that counts apples in your apple picking machine.
What we sometimes see is that a backbone is trained using transformers, then a smaller CNN is destilled from that (DinoV3 for example).