ActorNightly
Joined 7 years ago
Joined 7 years ago
Gemma4:31b, which is the better model with more complex architecture (and thus can't be compiled into a megakernel), is gonna be way slower in prefill, and ~50% slower than a 3090 even with oMLX. You can't get away from much slower shared ram vs vram, and less parallelism compared to 3090.
The other advantage of an eGPU setup, you can simply add/swap performance. Dual 4070s give you 24b of ram and you can still get about 80% of the performance of a single 3090, and actually you get faster prefill due to more parallelism.
So in terms of money/usefulness, Mac always loses.
It was done plenty. Just with extensions to python like numpy and opencv. There is no reason to translate everything versus just the critical pieces.
Try more like 70-130 tok/sec. https://github.com/L-Forster/open-jet/tree/master/megakernel.
And also Qwen models famously chase benchmarks. They are good for writing code, but fall behind in other areas. Gemma4:31B is 6gb larger and does a much better job across all areas. https://www.reddit.com/r/LocalLLaMA/comments/1vyzopv/gemma4_...
Go ahead and run Gemma4 on your M1, and you will find that its unusable at 12 tok/sec.
And this is also not touching prefill, which is the biggest thing that you need fast, and Apple sucks in this regard compared to nvidia cards. With gfx cards, you can throw big codebases at the local models and it will happily crunch through them in very little time.
Sure, spending 1.5k on a Macbook enables you to run a local model, and if you really have a good reason for running local (like running heretic/abliterated models) for tasks you don't mind waiting for, it works. The difference is, spend 1k more and let you never have to pay for tokens, and you can run inference all day long without your computer thermal throttling, like your Macbook.
And I don't need to make assumptions when you say idiotic stuff like "We're getting close to local AI being really useful", like you have any idea what you are talking about. On par with being an Apple fan though.
Mini PC with Occulink, EGPU dock, 750w power supply and a refurbished 3090 is 2500. It can run Gemma4:31b at 1000+ tok/sec prefill, ~40 tok/sec generation (which is slow, but a fully connected model is way better for all tasks)
if you want more specialized partial activation models like Qwen, you can get 100+ tok/sec generation.
This beats anything that Apple has by a long shot. You don't need to run large models for usefulness. Large models can get things write the first time, but thats their only advantage. Running small models that are good enough like gemma4:31b fast allows you to do faster iterations.
And if you really want, you can also do multi GPU setups if you want slightly slower model loading with usb4/thunderbolt ports.
TLDR; local ai has already been useful for quite some time if you arent brainwashed into believing that Macs are the best for running local inference.
Dunno how much investment is needed into language optimization anymore.
https://news.ycombinator.com/threads?id=ActorNightly&next=49...
TLDR its not so much about the transformer, as much as expanding dimentionality. You can chose any which way to do so.
Hate to burst your bubble, but behind Trump there are quite a few people that are ready to take his place, with no real chance of voting ever being able to stop them. Don't get your hopes up that voting blue matters. Everyone had the chance in 2024, half the eligible population did not vote. Now we get to live with those consequences.
US is done for.
There are 3 big factors in play
* 2020 attempted coup gave the Republicans valuable information on what to do this time around. Basically, the game is to put US into legal limbo and remain in power. They know nobody on the democrat side is going to think about using armed forces, and they are not going to use armed forces to keep their public image and keep economy from panicking, so they are going to just declare some emergency and keep power under the pretense of fixing that emergency. Reminder that Supreme Court is on his side.
* Trump has shown that he has no issue with having a private army operating with extrajudicial authority in public. Now imagine with what he is ok doing with a non-public army. You should fully expect people going against him to get "arrested" under some made up charges.
* Even with Trump gone, there are smarter, younger, non senile conservatives ready to take his place and grab power and do the same things, with way less mistakes. Vance is right up there with him.
As such, even if everyone gets out and votes blue, in the end its all pointless. Trump can get impeached, convicted, and either Supreme Court says nah, or he just won't leave. In 2028, he (or Vance) will declare election fraud again and do it successfully this time to remain in power.
And even if somehow Dems win the house and presidency, and manage to pack supreme court to take control of all the branches, the thing is the will be left with a insanely shitty situation to clean up that will result in economic downturn, and the idiots in US will vote republican again in 2032.