lxe

That's because every single model release after GPT-3.5 has been, at least in part, been trained on data synthesized by GPT-3.5 and then the other OpenAI models, as well as Anthropic models later.

Also the distillation attacks and other distillation methods that gather chain of thought information from frontier models are also used to generate vast amounts of data in order to generate synthetic datasets to post-train what I believe is the vast majority of models that are being released.

If you look at any information about how these models are trained, the details about how the data has been synthesized, where the agentic traces come from that are used in post-training, etc., they almost never tell you about exactly what model was used to generate them.

Which I don't understand. We have wide screens. I want to stack a lot of columns on them.

I was on low-dose Ativan for a few weeks casually prescribed for anxiety. What followed is an absolute year of hell. The most unimaginably difficult time in my life. Usually they tell you that short-term low dose poses lower risk or protracted withdrawal, but it's not a purely statistical risk. There are potentially pharmacological, physiological, and other factors that govern how these drugs affect each individual and those are largely unknowable.

After Iosevka came out, no other coding font has replaced it for me

Interference optimization is a solved problem with no moat. Anybody can achieve this result, optimize for a particular hardware, optimize for a particular problem. In general, software is no longer a moat.

The moat is custom inference hardware, lower power, higher throughput, etc.

I specifically tuned it for maximum heat and noise

Hello it is I, lxe

Perplexity looks bad and unsloth just did some good quants but they don’t perform as good on my asymmetric old dual 3090 box.

However, actual quality evals show literally zero degradation in “hard math” questions against full hosted model at a cost of 1.5x - 2x CoT tokens. I expect other quants to suffer from this too. This behavior seems to decrease as precision increases. I also suspect it’s the specific tuning a what experts I’m picking.

Nothing is stopping it. This model is woefully bad at accurate creative red teaming however. GLM finetunes on the other hand are pretty good. And I'd bet they are already deployed and doing all sorts of deeds.

Why isn't this type of expert caching in the native llama.cpp yet? Why do we need a separate codebase?

Hah, but of course the official documentation will state to launch it in "set it and forget it" mode by defining the ultimate goal. This way it's guaranteed to waste maximum amount of tokens.