... which is a single JAR that does SIMD accelerated inferencing of local models, dependency free. Everything is written in Java from the tokenizer up. It can also use Metal on macOS. Small models can be surprisingly powerful these days and all you need is a fast CPU. It works not only for LLMs but also image models, video models and speech.
Remarkably, it is significantly faster than llama.cpp on some tasks like prefill (decode is memory bandwidth constrained so language hardly makes a difference). At least as long as you run on GraalVM (JVM or native image, both work). So you can actually beat C++ performance for AI in Java.
https://qxotic.ai/
... which is a single JAR that does SIMD accelerated inferencing of local models, dependency free. Everything is written in Java from the tokenizer up. It can also use Metal on macOS. Small models can be surprisingly powerful these days and all you need is a fast CPU. It works not only for LLMs but also image models, video models and speech.
Remarkably, it is significantly faster than llama.cpp on some tasks like prefill (decode is memory bandwidth constrained so language hardly makes a difference). At least as long as you run on GraalVM (JVM or native image, both work). So you can actually beat C++ performance for AI in Java.
It's impressive stuff.