Comment by tikkun

For context: based on current gpt-4 times,[1] GPT-4 is about 12 tok/s on avg. GPT-3.5 is about 35 tok/s. OP got 13 tok/s on an A100. And 10 tok/s on M2 Ultra.

[1]: This site that I run https://openai-status.llm-utils.org

Step by step guide: https://mlc.ai/mlc-llm/docs/get_started/try_out.html