edunteman

Nice work! Had to do similar serverside for stripping named entities and ended up burning way too much LLM cash

I hate to be that guy, but you can just choose to do these things. Learn to set SMART goals. Then set just one, so there's no excuses around priorities. Obviously, choose well. Make that goal your life.

I’ve been a big fan of “what’s the thinnest this could be” interpretations of sandboxes. This is a great example of that. On the other end of the spectrum there’s just-bash from the Vercel folks.

The llm detector in my brain went off too

Exactly, that’d be the intention. For compute-heavy or long running jobs you’d still probably want a dedicated VM like on E2B but for quick stuff, bVisor

Correct, your PATH resolves to your local tools as if it was unprotected bash, but syscalls are filtered/virtualized

Hell yeah, love to hear it! Happy to answer any questions or issues you run into

The part that most resonates with me is the lingering feeling of “oh but it must be my fault for underspecifying” which blocks the outright belief that models are just still sloppy at certain things

Good question, I imagine you’d need to set up an ngrok endpoint to tunnel to local LLMs.

In those cases perhaps an open source (maybe even local) version would make more sense. For our hosted version we’d need to charge something, given storage requirements to run such a service, but especially for local models that feels wrong. I’ve been considering open source for this reason.

Ah yes that makes sense, have heard of those cases too but hadn’t put much thought into it. Thanks for pointing it out!

I’d love your opinion here!

Right now, we assume first call is correct, and will eagerly take the first match we find while traversing the tree.

One of the worst things that could currently happen is we cache a bad run, and now instead of occasional failures you’re given 100% failures.

A few approaches we’ve considered - maintain a staging tree, and only promote to live if multiple sibling nodes (messages) look similar enough. Decision to promote could be via tempting, regex, fuzzy, semantic, or LLM-judged - add some feedback APIs for a client to score end-to-end runs so that path could develop some reputation