Comment Comment by behnamoh I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformersThey claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations."AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0].What's weird is that MLX is made by Apple and yet, they can't support ANE given its closed-source API! [1][0]: https://github.com/ggml-org/llama.cpp/issues/10453[1]: https://github.com/ml-explore/mlx/issues/18#issuecomment-184... parent behnamoh 1 year ago
They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations."
AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0].
What's weird is that MLX is made by Apple and yet, they can't support ANE given its closed-source API! [1]
[0]: https://github.com/ggml-org/llama.cpp/issues/10453
[1]: https://github.com/ml-explore/mlx/issues/18#issuecomment-184...