Author here. I'm a big fan of factories but the slowness is a real drag on large test suites. If you're considering switching, remember that you can do it gradually, there's no law against using both fixtures and factories in the same project, in some cases (mostly on very complex domain data models) even makes sense: fixtures for the base setup that all tests share, factories for additional test specific records.
Btw, I also have an article with some of my learnings using factories and I make a remark on how it helps with test speed: https://radanskoric.com/articles/test-factories-principal-of...