chisleu

Does it still have rediculous system prompt alignment preventing use on red teams?

It is and it isn't. Why are you comparing the m5max instead of the m4ultra?

The big deal to me is the number of compute cores for prefill tps, which is suppose to be 4x faster on the m5ultra.

It's my opinion that the m5 ultra is going to be a really big deal in terms of local AI accessibility. Flash sized models (~200-300b params) are going to be reasonably fast as long as you aren't throwing 40k context at it on each or the first request (ie, agentic harnesses).

Even agentic harnesses like Cline should move at a reasonable clip on m5 ultra. I suppose we will know sooner than later.

FYSA: Former m4 ultra 512GB owner and current 4x rtx6000 owner here. I upgraded because I needed more prompt processing speed and concurrency.

Many are investing in RTX6000pro hardware but hitting deployment walls. This guide covers everything from simple 4x builds to scaling up to 16 cards with PCIe switches.

An open-source, code-first Go toolkit for building, evaluating, and deploying sophisticated AI agents with flexibility and control.

Total tangent, but I got to ride in some of these on a recent trip to India and I was really impressed with the build quality and utilitarian usefulness of the design.

Because of the prompt processing speed, small models like Qwen 3 coder 30b a3b are the sweet spot for mac platform right now. Which means a 32 or 64GB mac is all you need to use Cline or your favorite agent locally.

I've been using GLM 4.5 and GLM 4.5 Air for a while now. The Air model is light enough to run on a macbook pro and is useful for Cline. I can run the full GLM model on my Mac Studio, but the TPS is so slow that it's only useful for chatting. So I hooked up with openrouter to try but didn't have the same success. Any of the open weight models I try with open router give sub standard results. I get better results from Qwen 3 coder 30b a3b locally than I get from Qwen 3 Coder 480b through open router.

I'm really concerned that some of the providers are using quantized versions of the models so they can run more models per card and larger batches of inference.

This is going to improve the quality of LLM responses for users. I'm for this.

> but eventually they did it

They can do this with manual partitioning indeed. I've done it before, but it's not ideal because the auto partitioner will scale beyond almost anything AWS will give you with manual partitioning unless you have 24/7 workloads.

> you can be throttled for more than a day before it kicks in

I expect that this would depends on your use case. If you are dropping content you need to scale out to tons of readers, that is absolutely the case. If you are dropping tons of content with well distributed reads, then the auto partitioner is The Way.

and indeed the bucket is not separate from the object key. the API separates it logically "for humans" but it's all one big string

> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.

As of when? According to internal support, this is still required as of 1.5 years ago.

/agree

We are in the infancy of LLM technology.