Comment by johsole

What kind of hardware are you using and what is your token generation speed (tok/s)? Every time I've tried to run local models, even on my beefy mac book (128Gb Ram), I've been very disappointed at tok/s speed.