Comment Comment by Aurornis > Cloud hardware is not inherently more "proper" than what's being proposed hereCloud hardware can run the original model. Quantization will reduce quality. The quality drop to Q4 is not trivial.Cloud hardware is also massively faster in time to first token and token generation speed.> there's nothing wrong per se about targeting slower inference speeds in a local single-user context.If that's what the user wants and expects then it's fineMost people working interactively with an LLM would suffer from slower turns. parent Aurornis 6 months ago
Cloud hardware can run the original model. Quantization will reduce quality. The quality drop to Q4 is not trivial.
Cloud hardware is also massively faster in time to first token and token generation speed.
> there's nothing wrong per se about targeting slower inference speeds in a local single-user context.
If that's what the user wants and expects then it's fine
Most people working interactively with an LLM would suffer from slower turns.