Comment

Comment by _davide_

By design it can't be significantly slower than Jev: the prompt processing (AKA PP) is exactly the same on both and will take most of the time. Then you can process every single "question" in parallel, just predicting one or two tokens (if an answer is ambiguous with a single token) per each question, again in a single batch.

So, fast in the LLM space and comparable with Jev.

parent _davide_ 1 month ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton