Is there a tutorial on how to get that setup running step-by-step? I only found a GitHub issue (https://github.com/ggerganov/llama.cpp/issues/4439) that mentions that mainline llama.cpp isn't working for the model.
Bonus question if you have the time: there's a release by TheBloke for this on HuggingFace (TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF); but I thought his models were "quantised" usually - does that kneecap any of the performance?
Bonus question if you have the time: there's a release by TheBloke for this on HuggingFace (TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF); but I thought his models were "quantised" usually - does that kneecap any of the performance?