wishawa

I see. That's slightly unfortunate. In principle, increasing temperature flattens out the distribution but the ordering between different tokens' probabilities remain the same, so setting a top-k shouldn't break my test. Can't say the same for top-p though. And all of this is probably too deep into the provider's implementation details for me to make assumptions on.

This is with Together's API via OpenRouter, running DeepSeek V3 0324 and Kimi K2 0905.

I didn't set a top-k. So it seems like Together must be doing something weird in their speculative decoding implementation.

I didn't know this! I've always thought speculative decoding was "if p(draft_token) > threshold, use it". You made me go read how it actually works and it's pretty neat!

That said, I still think some providers are cheating. Please correct me if the test below is flawed.

I generated texts at temperature = 0 vs temperature = 2. At high temperature, the distributions effectively become flatter, meaning the difference between real and draft effective distributions (the D_LK used in theorem 3.5 of 2211.17192) becomes smaller. When T=2, the model speaks complete gibberish, so the effective distribution must be pretty flat. This should mean fewer rejections --> a lot faster speculative decoding. Yet, I see no increase in throughput at all...

Fair point. If Moonshot is holding back the true weights or inference techniques that affect correctness, then providers including Together should call them out on that. I for one would stop using Kimi if that is the case.

Anyway, Novita is doing significantly better on the vendor verifier chart than Together, so the low quality must be partially Together's fault at least.

Euclidea?

The instruct model is trained on the Baize dataset, which is generated from ChatGPT. So it is no surprise that Falcon censors the topics that ChatGPT censors.

If you have an uncensored instruct dataset, you can use that to fine tune your own uncensored version from the base Falcon.

You're right. I got the two frameworks confused.

Async UI is similar to Sycamore in term of not diffing VDOM.

The API is different in that in Sycamore, you tell the framework what to render (by using sycamore::render) and the framework will handle it from there. In Async UI, you await what you want to render yourself. Async UI's API is more transparent in this way, and this brings benefits including - making async control flow (like the control flow example in the blog post) possible - making component simply async function or anything that implements IntoFuture<Output = ()>

Sycamore's reactivity is pretty painless (a little too magical for my taste, but that's probably just me), so it's something Async UI can learn from.

I must admit I've never seen Concur before!.The idea of using generator/async is pretty similar. The difference, as I understand, is that in Concur, you yield the widget and let the framework mount it, while in Async UI, you await the widget yourself.

The Todo example is 2.5 MB non-gzipped. About 600KB gzipped.

Still large, but it's because we're shipping a lot of code that we don't need. Mostly the APIs exposed by web_sys. Proper dead code elimination would bring it down a lot.

By lots of 'static and Rc.