In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good:
I mention this because it could theoretically be applied to Mistral Moe. If the uplift is the same as regular Mistral 7B, and Mistral Moe is good, the end result is a scary good model.
This might be an inflection point where desktop-runnable OSS is really breathing down GPT-4's neck.
https://huggingface.co/fblgit/una-xaberius-34b-v1beta
https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16
I mention this because it could theoretically be applied to Mistral Moe. If the uplift is the same as regular Mistral 7B, and Mistral Moe is good, the end result is a scary good model.
This might be an inflection point where desktop-runnable OSS is really breathing down GPT-4's neck.