Comment Comment by brucethemoose2 I can confirm this, mlc is shockingly fast on my RTX 2060.The catch is:- MLC's quantization is somewhat different (though I havent run any perplexity tests yet)- There is no CPU offloading (or splitting onto an IGP) like Llama.cpp yet (unless its new and I missed it). parent brucethemoose2 3 years ago
The catch is:
- MLC's quantization is somewhat different (though I havent run any perplexity tests yet)
- There is no CPU offloading (or splitting onto an IGP) like Llama.cpp yet (unless its new and I missed it).