anerli

How times have changed! Working on agents for so long helped us understand how inference needs to work to actually support local agents properly.

Currently it's not something our engine supports, but in theory our kernels could help for any training or fine-tuning jobs as well. If it's something you'd be interested in, feel free to create an issue on GitHub describing your use case!

Hey, yeah this is a known issue on M5+ macs. We are working on a patch so that our kernels use that hardware acceleration path. This should make prefill faster than MLX-based engines and boost decode a bit more for that hardware!

We have a couple bugs we are patching where we are reserving too much memory overhead, you should actually be able to fit a couple different models on there!

Plus in the near future, we'll add ways to automatically utilize your RAM as well (such as expert-streaming).

Our goal is to make the best use of whatever hardware you have, even if its not high end!

Yes :) same concept could apply there. If it's something you're interested in feel free to open an issue in our GitHub!

Hi, M5+ Macs have a known optimization gap since we do not fully utilize the Metal 4 matmul operations in our kernels yet - so should be able to do much better on this specific comparison soon!

Hi, the model ranking scores are based on whatever machine Magnitude is actually running on. So it will account for your specific memory capacity and performance characteristics to recommend appropriate models. The estimator is purely to help filter and recommend a model. The tuning is the only part that actually effects real performance, and is done automatically whenever you download a new model.

Yes MLX is generally a better comparison point overall for Apple, planning on releasing a benchmark for that soon. However against the MLX-based engines we've compared with so far Magnitude will continue to have an edge, especially for decode kernels.

Prefix cache is re-used with a prefix tree structure for maximal re-use across sessions sharing prompts.

The endpoint is standard OpenAI compatible chat completions.

Yeah that doesn't sound quite right. Given that the 5080 has 16 GB of VRAM I would have expected a few more options, for example Gemma 12B, to at least show up as available. Do any bigger models show as available or was that specifically the recommendation?

The MoE 35b might be tight though unless you were to go below 4-bit. Could you share the quant you used when you ran this on that 5080 before? Our catalog only contains models down to 4-bit because we find that thinking, tool calling, and overall capabilities start to suffer at lower fidelity.

Feel free also to create a GitHub issue with more details and we can take a closer look.

In the short term we will enable using multiple devices within one machine efficiently and distributing the workload of a model between them. If you mean running the same model distributed across different devices, yes we do have plans to enable that. You should eventually be able to run larger models across machines with capable hardware as long as you can provide a fast enough connection between them.

Core AI is a way to ship a model inside an app rather than an inference engine you can point agents to, and it's not meant for agent workloads.

Once you get down to the level of writing GPU kernels the OS isn't very much in the way, aside from some host-side memory logistics.

So it's more about making those GPU kernels perform the required memory move and arithmetic operations as close to the theoretical optimum as possible.

The goal of the inference engine is to make the best use of whatever hardware you have to run models performantly and let people run bigger models. At first this will include using all the hardware on a given machine optimally. Eventually we also want to support interconnect between multiple machines to enable running bigger models!

Will be adding this one very soon!