theapadayo

This headline could literally be a NileRed video.

Not just structured output. Dropping down to logprobs, prompting the model to emit one word as the answer, and then ranking the output tokens to pick your answer works great on small Qwen & Gemma models.

The fascinating part to me is that Jev seems like this technique plus post-training to get multiple independent confidence values for each possible answer.

Yeah encryption isn't the issue. The only way I see to fix this is if you stop the user from switching models mid-session, or strip out the thoughts when switching models. Either way you're degrading the user experience.

IMO the biggest thing still missing is an actual way to define the model architecture outside of being hard coded into the current build. It doesn't need to be a 1:1 performance parity with the fully supported models. Having proper, vendor validated support for day 1 is what is the difference between people thinking a model is amazing vs horrible. See recent Gemma vs Qwen releases.

Not sure what the solution is, other than writing a DSL to describe the model graphs which you then embed in the GGUF. The other fallback is to just read the PyTorch modules from the official model releases and convert that to GGML ops somehow.

> TFS (what microsoft had/pushed before acquiring github)

It's still around. It's just called Azure DevOps now. I personally think it's great for what it does.

Well I guess we finally got the mythical 'Q*'. Or at least some variant of it using energy functions (I think that's what they mean by 'soft' Q-learning?). The extra boost from using the value function at test time is interesting as well.