alexeldeib

It's a button

What issues do you see in practice? This seems pretty easily "fixable"

Why are those incompatible? Pricing is 10% under moonshot, and pure infra providers also want money

KV cache is, well, a cache that can fill up and trigger eviction. You require enough space to execute at least 1 fwd pass of 1 request at your context length. KV cache hits reduce TTFT by avoiding prefill. You don’t get to skip decode.

MoE is kinda related in terms of lower usage requirements vs a dense model of same total param size, but I think your mental model is a bit off.

“It depends”: what’s your prior experience, what kind of roles interest you, how big is the gap between what you have + a little ML knowledge/side projects?

I’d argue there’s a big need for people with solid fundamental CS, sysadmin, infra skills who can bridge the gap into ML practitioner/researcher understanding. Applications or inference generally are probably easiest to break into, especially if you already have service knowledge. If you want to work on distributed training or kernel/model optimization, you probably need to prove your chops there.

Neoclouds, startups in the AI space, maybe hw vendors are probably good places to look.

as someone in the space this ticks a lot of boxes: kubernetes-native, strong isolation, python sdk (ideal for ML scenarios). devmapper is a nice ootb approach.

Glancing at the readme, is your business model technical support? Or what's your plan with this?

Anything interesting to share around startup time for large artifacts, scaling, passing through persistent storage (or GPUs) to these sandboxes?

Curious what things like 'Multi-node cluster capabilities for distributed workloads' mean exactly? inter-VM networking?

Oh nice! I just had an excuse to try mojo via max inference, it was pretty impressive. Basically on par with vllm for some small benchmarks, bit of variance in ttft and tpot. Very cool!

ouch, last “recent update” in 2023. Any idea what happened?

Larger memory, weaker comms. You can optimize for this by doing things like increasing batch size/data parallelism vs sharding schemes with more comms.

At scale training won’t be able to avoid comms entirely, while many models can fit in a single MI300 for serving.

yeah, the system/application distinction feels somewhat superficial. The “multiple user space” inside a container thing sounds interesting (not sure what that means exactly), but maybe more similar to a Kubernetes pod, except maybe instead of different rootfs there’s another isolation mechanism?