Comment by ireadmevs

Try checking what M-x magit-toggle-verbose-refresh has to say, you may be able to identify what's slow and optimize around it