charles_irl
Joined 2 years ago
Karma
371
Submissions
63
Member since
Aug 6
Building useful technology out of large neural networks. https://modal.com
View on Hacker NewsRecent submissions
Comment · 1 month ago
Thanks!
We chose the Qwen model for the initial and core results because it's a simple architecture, which made the roofline cost modeling very easy and trustworthy. I'd already done related work with that model before for another project on speculative decoding, so we had a reference impl as well.
For brevity, this post focuses on a subset of the results. In the companion post on the Full Stack Data Lab blog[1], we go into more detail. With any luck, that'll be a nice VLDB-bound arXiV paper soon!
We have some more results on DiffusionGemma that we tacked on after Jev started popping off. Hard to say much about the specific Jev model, since the details are not known -- even to me, working with their team as they scale up on our platform.
But generally, I think a Quail-like query planner-based approach is a slam dunk for this broad set of classifier models. Quite likely that the benefit is primarily for backend/analytics queries though, where the superior KV management has a chance to shine.
2B tokens/minute/H100 happens to be roughly the speed of light for Qwen 4B. Probably unattainable, for the reasons we indicated (and others, e.g. manufacturer overstatement of peak rates), but I wouldn't be surprised if we could squeeze another 20-30% with CUDA PDL or a megakernel.
Comment · 5 months ago
Very cool!
> Unless you've actually watched tokens stream at those rates, the numbers are hard to internalize. This is the rendering.
I built something similar recently, for the same reason: https://modal.com/llm-almanac/token-timing-simulator.
I like that the output rendering is closer to typical UIs -- syntax highlighting in code mode, tool calls, dim-italic reasoning.
One feature mine has that the author, or anyone else who vibe codes their own version after seeing this, might like to steal is modeling the distribution of output latencies. My implementation is hacky (log-normal roughyl estimated from p50, p90, and p99 values), but still, when you set those to realistic values, it recreates the "jitter" you see in many LLM UIs.
antirez is right that generation tok/s isn't flat as a function of context length, which is a weakness of both simulators.
Comment · 5 months ago
To clarify: we do content-based hashing, and when we say "shared bytes aren’t guaranteed to be in the exact same container image layer", what we mean is that
FROM some/image RUN pip install torch==2.7.1
and
FROM another/image RUN pip install torch==2.7.1
will produce images with very high overlap in contents, which will be shared by a content-based cache, but those images' final layers are disjoint from the perspective of a layerwise cache.
Comment · 5 months ago
You're absolutely right!
Comment · 5 months ago
Yep! That should start in ten seconds or so -- about a second per gigabyte of weights, plus a second to start the container and a few seconds to load the memory snapshot.
There are a few limitations with snapshotting, e.g. it generally fails when using multiple GPUs, which we document here: https://modal.com/docs/guide/memory-snapshots.
Comment · 5 months ago
Cutting latencies by 40x! Unfortunately couldn't fit the whole title in the character limit :<
Comment · 9 months ago
> Ed Zitron also called out the business model of GPU-as-a-service middleman companies like modal deeply unsustainable, and I also don't see how they can make a profit if they are only reselling public clouds.
You got a link for that? I work on Modal and would be interested in seeing the argument!
We think building a proper software layer for multitenant demand aggregation on top of the public clouds is sufficient value-add to be a sustainable business (cf DBRX and Snowflake).
Comment · 9 months ago
Sorry to lead with a bunch of jargon! Wanted to make it obvious that we'd give concrete recommendations instead of palaver.
The technical terms there are later explained and diagrammed, and the recommendations derived from something close to first principles (e.g. roofline analysis).