anemll
Joined 1 year ago
Karma
309
Submissions
27
Member since
Feb 12
Recent submissions
Comment · 14 days ago
It's warm but not hot. On M6 it's about 50% less energy consumption in my tests vs GPU. I think iPhone should be similar
Comment · 1 month ago
Not all systems affected
M1 and M5MAX are OK
Comment · 7 months ago
Check it out,
you might be able to speed it up using this
https://github.com/Anemll/anemll-flash-mlx
https://x.com/anemll/status/2038684375425200360
Comment · 7 months ago
17B includes 10 expert plus one shared. So actual size of the expert is much smaller
Comment · 7 months ago
Check my repo, I had added some support for GUFF/untloth, Q3,Q5/Q8
https://github.com/Anemll/flash-moe/blob/iOS-App/docs/gguf-h...
Comment · 7 months ago
Thanks for posting this, that's how I first found out about Dan's experiment!
SSD speed doubled in the M5P/M generation, that makes it usable!
I think one paper under the radar is "KV Prediction for Improved Time to First Token" https://arxiv.org/abs/2410.08391 which hopefully can help with prefill for Flash streaming.
Comment · 7 months ago
SSD streaming to compute units is new.
M4 max can do 15 t/s with its 15GB/s drives
Comment · 7 months ago
Yes, SSD speed is critical though. The repo has macOS builds for CLI and Desktop.
It's early stages though. M4 Max gets 10-15 TPS on 400B depending on quantization. Compute is an issue too; a lot of code is PoC level.
Comment · 7 months ago
multiple NAND, and apple already used it in Mac Studio.
Plus better cooling
Comment · 7 months ago
both, tbh
Comment · 7 months ago
Probably 2x speed for Mac Studio this year if they do double NAND ( or quad?)
www.anemll.com https://github.com/anemll https://huggingface.co/anemll https://github.com/anemll