paguasmar

Great question. The controller provision a replacement automatically. The controller runs continuous health-check polls against the worker’s, if it doesn't respond, provisions a new one.

Why not use a AASG?

I chose to use a controller to make the solution cloud agnostic.

You're right. A CPU only model is definitely is not a real prod workload where you'd need GPU machines.

Here is a rough table of model size to instance type to spot $/hr:

For up to 8B, you can use a c6i.xlearge which costs as spot about $0.4/hr.

For up to 70B, you can use a g5.12xlearge that costs you about $2/hr.

Besides the GPU machine be aware that you need a controller instance to route the requests and scale. That has a fixed cost of up to $0.08/hr. When not in use at all, just type veloxml down --all and it tears down the controller too for true $0/hr.

We're currently running larger LLM models benchmarks to add to the README this week.

Basically the CLI

1. compiles your model code and dependencies without requiring Dockerfiles or K8s

2. provisions spot or on-demand instances directly in your own AWS/GCP account (via SkyPilot)

3. it then spins up the runtime and gives you a production-ready URL endpoint

I wrote this originally to replace Modal and Baseten, which I wasn't too pleased with. I needed to deploy open-source models inside my own VPC without vendor lock-in or proprietary Python decorators.

Thought the community here might like it. AWS is an amazing cloud - with one-command deployment and scale-to-zero I was able to consolidate all my LLM serving into my own infrastructure for a fraction of the cost.

Would love feedback. There are a few (many?) bugs and a lot of things to iron out, but I'm working with some close friends to make it awesome.

I like the project. I see a lot of potential integrating it with LLM providers in an effort to lower token usage for repetitive tasks. Your solution becomes the "main model" and the LLM the fallback

The UI looks good and aligns well with your retro positioning.

Open-source the project is smart to allow the community to build plugins and have fun with the device.

It's great that you found a way to simplify the extraction algorithm.

Can you share the PC specs you used to run the test for both the WinZST and 7-Zip experiment?

I see a lot of potential hearing my morning email podcast while driving to work.

For folks with kids or driving other people in the car it might be embarrassing reading your sensitive emails out loud, such as dating apps or medical diagnosis.

Do you have a check that drops sensitive emails from the podcast?

Avoiding MELCloud is the dream.

I'd love to experiment this with my Mitsubishi R32 split. Have you tried it on Mitsubishi Heavy Industries models?

Within 10 secs of reading this I was convinced I needed this in our pipeline.

For me the next step is plugging this into the company CI/CD to gate SKILL updates and catch tool call regressions before merging

Gotta catch 'em all!

For me the next step would be picking a 6 Pokémon squad and generating a combined type cov and weakness matrix