Comment by cma

May still happen especially if it is thrashing vram in and out of system memory or something, but have you tried lowering priority of the stable diffusion process?