Comment by roadside_picnic

> from 40 to 50 tok/s generation

That's actually much better than I would have thought!

Thanks for the answer and it does make this approach make more sense as a budget solution to running larger models locally.