Comment Comment by kkielhofner llama.cpp and others can run purely on CPU[0]. Even production grade serving frameworks like vLLM[1].There are a variety of other LLM inference implementations that can run on CPU as well.[0] - https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#su...[1] - https://docs.vllm.ai/en/v0.6.1/getting_started/cpu-installat... parent kkielhofner 2 years ago
There are a variety of other LLM inference implementations that can run on CPU as well.
[0] - https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#su...
[1] - https://docs.vllm.ai/en/v0.6.1/getting_started/cpu-installat...