Comment Comment by theaiquestion Compete with Llama.cpp? Like transformers llama [0], exllama [1] (really fast), or litllama [2] ?exllama is really memory efficient and really fast[0] https://huggingface.co/docs/transformers/main/model_doc/llam...[1] https://github.com/turboderp/exllama[2] https://github.com/Lightning-AI/lit-llamaEDIT: Or do you mean cuda? Because yeah, it's such a shame AMD's Rocm is so bad even geohot gave up. it's examples don't even run without crashing.https://github.com/RadeonOpenCompute/ROCm/issues/2198#issuec... parent theaiquestion 3 years ago
exllama is really memory efficient and really fast
[0] https://huggingface.co/docs/transformers/main/model_doc/llam...
[1] https://github.com/turboderp/exllama
[2] https://github.com/Lightning-AI/lit-llama
EDIT: Or do you mean cuda? Because yeah, it's such a shame AMD's Rocm is so bad even geohot gave up. it's examples don't even run without crashing.
https://github.com/RadeonOpenCompute/ROCm/issues/2198#issuec...