I moved to using Fastmail as my primary over 7 years ago and I still use it as my main email address but I had hoped I could get rid of my gmail but instead it became a catch-all junk inbox similar to OP's gmail. The main issue is that so many websites use Login in with Google as a way to sign-in that I just had to keep my gmail logged in most of the time.
So instead of "moving away from Gmail to Fastmail" it's more like "using Gmail and Fastmail" for me.
So instead of "moving away from Gmail to Fastmail" it's more like "using Gmail and Fastmail" for me.