Appreciate your response.
We compared MKML mk600 (5.2GB) against llama.cpp Q5_1 (4.7GB) and Q6_k (5.1GB) on a 4090 for llama-7B. The test is the same in all cases: we generate 128 tokens from a single token prompt (batch=1) and measure performance of the forward pass during auto-regression.
(llama-7B, single prompt, batch=1)
MKML mk600: 125t/s
Llama.cpp Q5_1: 8̶4̶ 128 t/s
Llama.cpp Q6_k: 7̶8̶ 116 t/s
Our llama.cpp test:
Build (https://github.com/ggerganov/llama.cpp#cublas):
make -j12 LLAMA_CUBLAS=1
Run:
./main -t 16 -ngl 3̶2̶ 35 -m llama-2-7b-chat.ggmlv3.q6_K.bin -p "?" -n 128
Please feel free to post your llama.cpp results if they are different.
>Maybe I'm misunderstanding MKML. As I understood it, MKML is a compression step that then feeds into another framework like HF's Transformers or PyTorch.
MKML is not a compression tool that feeds into another framework. It is an inference runtime (like FasterTransformers or vllm) except that MKML is also plug and play with existing frameworks like Hugging Face.
We compared MKML mk600 (5.2GB) against llama.cpp Q5_1 (4.7GB) and Q6_k (5.1GB) on a 4090 for llama-7B. The test is the same in all cases: we generate 128 tokens from a single token prompt (batch=1) and measure performance of the forward pass during auto-regression.
(llama-7B, single prompt, batch=1)
MKML mk600: 125t/s
Llama.cpp Q5_1: 8̶4̶ 128 t/s
Llama.cpp Q6_k: 7̶8̶ 116 t/s
Our llama.cpp test: Build (https://github.com/ggerganov/llama.cpp#cublas):
make -j12 LLAMA_CUBLAS=1
Run:
./main -t 16 -ngl 3̶2̶ 35 -m llama-2-7b-chat.ggmlv3.q6_K.bin -p "?" -n 128
Please feel free to post your llama.cpp results if they are different.
>Maybe I'm misunderstanding MKML. As I understood it, MKML is a compression step that then feeds into another framework like HF's Transformers or PyTorch.
MKML is not a compression tool that feeds into another framework. It is an inference runtime (like FasterTransformers or vllm) except that MKML is also plug and play with existing frameworks like Hugging Face.