gbxk
Joined 1 year ago
Joined 1 year ago
https://www.mikeayles.com/blog/on-chip-llm-kv260/
500 tok/sec is great but this would be 40x and it changes the kind of tasks that it could even be used for if you link it to fast tools too.
How about using these Cactus models?
Would it make sense for you to collab with those guys (1) for you to design a cheap but improved, commercialisable version of your $250 chip and (2) for them to tailor their runtime and quantizations to such FPGA hardware?
https://github.com/cactus-compute/cactus
Also have you thought about using a Alveo V80? Still not crazy expensive and could fit bigger models with same approach
Will merge that in after it passes all network tests on a clean/wiped instance.
Our native K8s support and exposition of K8s API also makes it friendly to devops.
Finally, our deploy/infra stack is lean and tightly fits in a single Ansible playbook, which makes it easy to understand and contribute to, letting you rapidly gain full understanding and ownership of the stack.
gVisor isolates containers by intercepting system calls in a user-space kernel, so it can still be vulnerable to sandbox escape via gVisor bugs, though not directly through Linux kernel exploits (since gVisor doesn’t expose the host kernel to the container).
Katakate also provides more than isolation: it offers orchestration through Kubernetes (K3s)
You could create a gVisor RuntimeClass in Kubernetes to orchestrate gVisor sandboxes, but that would require extra setup.
I'll also add to the roadmap whilelist/deny for container pulling.
Thanks!