danielhanchen

We made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc)

We do plan to do larger benchmark suites though!

Hey we did not remove the MTP for sizes above 8GiB - but yes for small GGUFs under 8 ish GiB, we removed the MTP module (IQ2_XXS and lower), because it's 500MiB to 750MiB in size, and on small 8 GiB machines, even 500MiB is needed.

As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that.

But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL

That wasn't our problem right? Gemma officially updated tool calling which we adopted

Ok no worries - if there are any future issues - feel free to message / make a HF issue - we'll fix promptly!

Also note its best to follow Gemma4's official sampling params since they evaled with it - dry multiplier sometimes works, but it actually screws up reasoning sometimes

We will investigate Ling!

Thanks for the support and to the community!

Hey :)

Thanks haha

Hey yes - if you could describe what the issues are - we will gladly fix them!

Hey sorry what are the problems that you're experiencing - we're more than happy to help fix them!

Oh thanks for sharing! We launched a Desktop app with fast diffusion, video gen, training support, inference + tool calling, web search, canvas, HTTPS secure remote access via Cloudflared, API / model swapping + more!

It works in [Windows, Linux, Mac, WSL] X [NVIDIA GPUs, AMD GPUs, CPUs]!

If there are any issues / suggestions - we'll be quick to fix / implement them!

Oh thanks for sharing! The llama.cpp PRs should generally be fine for now - I'm fixing a few small edge cases as well!

Very cool write-up and GitHub repo!