colorant

Hardware Requirements:

- 380GB CPU memory for DeepSeek V3/R1 671B INT4 model

- 128GB CPU memory for Qwen3MoE 235B INT4 model

- 1-2 ARC A770 or B580

- 500GB Disk space

With ~1000 input, the TTFT is ~10 seconds

The ipex-llm implementation extends llama.cpp and includes additonal CPU-GPU hybrid optimizations for sparse MoE

Yes, but the context length will be limited due to VRAM constraint

Prompt length mainly impacts prefill latency (FTFF), not the decoding speed (TPOT)

This is based on llama.cpp

>8TPS at this moment on a 2-socket 5th Xeon (EMR)

Yes, you are right. Unfortunately HN somehow truncated my original URL link.