Researchers love the ViT and all its varieties, but a reminder for those following at home that ConvNets still work and scale fine, depending on your requirements. For example, in "A ConvNet for the 2020s"[1], the authors are able to scale up ConvNets to the sizes of ViTs.
From the abstract: "Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation"
From the abstract: "Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation"
[1] https://arxiv.org/abs/2201.03545