crowwork

The goal of the project is to bring open ABI and FFI for machine learning systems.

- Stable, minimal C ABI designed for kernels, DSLs, and runtime extensibility. - Zero-copy interop across PyTorch, JAX, and CuPy using DLPack protocol. - Compact value and call convention covering common data types for ultra low-overhead ML applications. - Multi-language support out of the box: Python, C++, and Rust (with a path towards more languages).

Scale LLM serving with programmable cross-engine serving patterns, all in a few lines of Python

XGrammar is an open-source library for efficient, flexible, and portable structured generation. Bring 2x-10x speedup in grammar grammar-guided(JSON and CFG) LLM serving.

runs on qwen2 on iphone with 26 tok/sec and a OpenAI style swift API

2b model running at 20tok/sec on iphone, nice potential for future applications