boomskats

Interesting because again, the license Kimi shipped under [0] defines "Model as a Service" as

> giving a third party access to language model inference or fine-tuning (e.g., via API) in a manner that allows such third party to exercise meaningful control over the inputs, parameters, or training data

and then they have that clause around if you operate such a business above $20M aggregate revenue you need a separate agreement with Moonshot before commercial use, which presumably captures the majority of the larger neoclouds best placed to optimise this.

But where's the line? Say you offer infra optimised for GPU inference, warm pools, isolation per customer, exposed control plane, billed per GPU-hour rather than token, the invoice says compute rather than calls. The customer arguably 'self-hosts', you're probably fine? And if you as a provider run the serving stack and hand your customer an inference API, you're inside the definition regardless of whether you charge by the second or by the token. But what about if you give them direct hardware access, but have the weights cached on NVMe / ramdisk hyperlocal to the infra they're renting so that their hf cli pull only takes a few seconds? Sure, a managed warm pool of GPUs with K3 pre-loaded probably isn't ok, but a local hugging face lru cache holding 'whatever your customers pull down most often', superoptimised for fast weight swaps that the customer controls... is? Is it?

Again, where's the line? Is it materially different from a local docker registry mirror? What about safetensors checkpoints pre-sharded for the specific hardware topology you're offering? Does it matter whether you perform the checkpoint optimisation yourself and make it available, or merely cache one published on HF that happens to target exactly the hardware you rent out? What if you published that checkpoint yourself?

I'm definitely overthinking this, and I'm sure there's been conversation here about this already, but the other kimi threads[1] are enormous. And I am curious.

I'm also curious to know whether Moonshot would actually be against a setup like this. Guessing they would if it was AWS (not quite elastic but not that dissimilar), but what about others? Realistically I guess it'd be easier to just talk to them, especially if you were doing it in a way that targets a slice of the pie they never would have gotten anyway due to data residency requirements etc..

After a bunch of middle clicking I landed here [0]. So if I understand correctly, the current stack switching proposal depended on exception handling to be implemented first for resume.throw, so that bit was blocked until now.

But it's also further along than you might assume [1], i.e. you can already invoke Wasmtime with `-W stack-switching` and hit the boundaries where the experimental implementation breaks.

Sadly though, like many ambitious Wasm things that don't have revenue directly attached to them, it now seems partly a case of finding someone willing to sponsor or finish the remaining work [2].

I also found the list of open stack-switching issues[3] useful.

(also, minor nit - Wasm != Wasmtime)