skohan

Be the change you want to see in the world

I think the 16" pro is already at the max battery size you can take on a plane.

Loading data to the GPU will generally never be the bottleneck. Yes that's a bit better for unified memory, but prefill and decode will still be bound by compute and memory bandwidth respectively.

There's some value in having a huge amount of memory addressable by the GPU for loading larger models, but even that doesn't make sense for current gen models. For MOE models, newer inference engines with smarter GPU expert caching and pinned memory allow a GPU with lots of bandwidth to significantly outperform a Spark, so that leaves you with larger dense models that fit on a spark but not on a GPU where you might have a use-case.

But even then it's hard. For instance a full-fat BF16 of a ~30B parameter model will fit on a 128GB spark and not a 32GB 5090, but at that size it's going to be frustratingly slow on the spark given the compute and bandwidth constraints.

Imo these devices are mostly good for experimentation, or if you only want to do overnight jobs or training which wouldn't be possible on a GPU, and you don't want to pay $100k for a DGX station.

Base M7 probably will still have pretty low memory bandwidth for serious AI workflows. M7 max/ultra might be pretty interesting though.

Linux is pretty great these days

Memory size is a bottleneck, but I have found that speed is very important for getting actual work done. Running a big model is great, but if you're getting 10t/s you're going to be spending a lot of time waiting.

And the cost of switching is near zero. It's not like trying to get SAP out of your org.

I'm also local-only in my current development process, and have a couple local agents running right now. The problem with local right now is the price has gone through the roof. Even 3-6 months ago I would have said it's viable for a lot of people, but even getting the RAM for a local inference machine is a big investment for a lot of people trying to dip their toes into the space these days.

The network effect is way weaker for LLM's. Switching from AWS to GCP or Azure is at least a multi-month project for an engineering org, if not completely operationally impossible. Switching LLM providers is at worst asking your engineers to install a new coding agent client, and at best switching a URL and API token in your config file.

Yeah it's a bit weird that it gets a different accent for part of the video.

But some people are not great speakers - I can imagine using AI narration if you don't feel like your own speech will meet your own standards.

> My tech career was just converting specs from clients and employers into machine code, with extra steps.

I guess the thing about that process is, you inevitably complete an incomplete spec. There are always fine details that will never be decided until they're required to complete the implementation.

I mean the labs are in a tricky spot where they have to justify massive valuations. "We're going to stay where we are and focus on inference" isn't the growth story you need when you're trying to justify a 2T valuation on 80B in revenue (unless you dress it up as a safety concern).

Google can afford to take their foot off the gas without killing their business.

No reason you couldn't in principle

You misunderstood my argument.

The point is, there are some domains (data-oriented programming) where Rust actively resists the kind of work you need to do to solve the problem in an optimal way, and it makes sense to use a language built around the type of memory management required.

There's no point in using Rust when 90% of your work is deactivating the benefits Rust provides and working around its default behavior.