Comment

Comment by ClarityJones

The Apple chips are quick when the model is "warm" on your second and subsequent requests. It's also quick if you don't have a harness, and feed "Hi, how are you?" as a prompt. However, a lot of common usage is to have a harness with lots of instructions and context. Loading that context is slow. So, the first response may take ~1 minute to return the first token (varies widely by hardware, model, etc.).
parent ClarityJones 2 months ago

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton