Comment by sheepscreek

For most folks in this thread running old M1 Max like me, look into MTPLX models (and the namesake inference engine).

I’ve built something custom that’s pretty darn fast but not ready for prime time yet. Might share it eventually.