richiejp

Joined 6 years ago

Karma

Submissions

Member since

Sep 26

I wrote privilege escalation exploit for a Linux n-day once

View on Hacker News

Recent submissions

Comment · 2 months ago

A microvm sandbox for AI agents because absolutely nobody else is working on that, but I am more insane than other people so that is my edge :-)

It boots an upstream Linux kernel with a config that can be defined by the user (doesn't take long to compile BTW, it's a small config). The workload runs in a container so it is compatible with devcontainers and stuff like that.

I use all the fancy stuff for observability e.g. eBPF gadgets and try to encourage agents to use these tools. All network and storage activity is monitored. It can all be blocked or recorded. It has a HTTPS proxy and network traffic can be recorded and played back from a content addressable store.

It can (in theory) send local directories to a remote sandbox, let an agent inside the sandbox make changes and then the user (or another agent) can inspect the changes made to the filesystem before accepting them back into the local directory.

It can be run as a cluster with multiple nodes, including a central orchestrator which has a web UI and I am currently implementing a terminal viewer, so I can run agents inside sandboxes then interact with them on the terminal via the web UI.

The entire thing is distributed as two self contained executables (at least it would be if the GitHub CI/CD was configured correctly).

I also included a bit of Sci-fi writing in my README. Also it is riddled with bugs, but I am really just seeing if it can work at all at this stage.

https://github.com/richiejp/darkaifactory

Comment · 3 months ago

Getting any performance improvement over vLLM is just a bonus although the RAM reductions are very nice. The main point is that we find deploying Python based projects to be a pain. I can't imagine how you know what files are unreachable in a dynamic language, but in any case that doesn't begin to solve issues like PyTorch requiring a C++ compiler at runtime to generate optimised CPU code. This isn't exactly embedded device friendly.

Having said that if you know of some magic bullet to fix the problems with the Python ecosystem (that vLLM and Pytorch are apparently unaware of) then I will be happy to try it because our CI is constantly blocked by Python related issues.

Comment · 3 months ago

I can only assume you are talking about LocalAI (not vllm.cpp) which is modular and contains adapters for many different AI engines written in various languages. LocalAI's core is written in pure Go and you choose what engines to install at runtime.

vLLM.cpp's runtime is pure C++, but AOT Triton compilation is done at build time in Python.