Comment by simonw

I ran Llama 2 70B in my browser (Chrome Canary on a 64GB MacBook M2) using this. It took a long time to start running, but then:

prefill: 0.9654 tokens/sec, decoding: 3.2589 tokens/sec

Honestly amazed that this is even possible. I haven't even run Llama 2 70B on my laptop NOT using a web browser yet.