thunderbird120

Lots of chiplet interconnects hurt your max performance pretty severely and drives up your complexity a lot. There's a reason they want to keep it monolithic.

I already harped on Cerebras, but their approach of just copy/pasting a whole bunch of identical functional blocks, over-provisioning the chip by ~8%, and then just fusing off blocks with defects allows them to effectively have 100% yield on a wafer-sized monolithic chip. This is very desirable, and just another reason I like their approach better.

This is a near perfect encapsulation of the exact type of person you should not take financial advice from.

Kimi Delta Attention (KDA), despite having "Attention" in the name, isn't really attention at all in any conventional sense. It's more like an RNN which can be efficiently parallelized during training. It's a very small modification to Gated DeltaNet, which can be described as an RNN whose hidden state acts like a small, editable attention memory.

Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.

Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.

RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.

>Curious to see if it holds up at frontier scale.

I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.

Medical expenses would be a combination of cost of insurance / treatment (PPP relevant) and government transfers in a country with partially or fully public healthcare, which is why it's so important to do both.

I see. That paper is looking a median income from a PPP perspective but without accounting for taxes and transfers, which is a tiny bit unorthodox in this exact context which tries to get into the weeds slightly more than just standard PPP adjusted GDP per-capita or something like that. It's valid statistically, but typically if you're trying to adjust for cost of living it makes a lot of sense to use the amount of actual income the individuals have left over after paying taxes and receiving transfers from social programs. Otherwise you'll have a number which accounts for differences in prices but not differences in the proportion of "income" which is actually retained by the earner. This is what that wikipedia article I linked is reporting. Comparing two different countries with very different tax/social program policies with PPP adjustments but without tax/transfer adjustments is less than ideal.

It's not wrong, it's just not very useful. It's typically a more intermediary statistic rather than a final one when doing this type of comparison.

Yes, since the weights being updated are a small subset of the overall total it's manageable. Just like how each separate conversation currently requires you to store a separate KV cache, you'd need to store the fast weights separately. Both KV cache and fast weight content stores have to be conversation specific, so just setting a bit of extra RAM aside for "memory" isn't really a new ask, just a different format for an old problem.

>Its turn around time is ridiculous, it has to be maintained with specialized equipment/hangers, along with external contractor assistance.

>Compared to the Gripen, as an example, which can land on a freeway and be up in the air again in a few minutes.

I have no idea where people got the idea that the F-35 requires a major refit after each sortie or that it needs climate controlled hangars, but there's literally no truth to any of it.

The turnaround time for an F-35 after a mission in a wartime scenario isn't going to be much different from any other older fighter jet. Refuel, rearm, get back in the air.

One of the key requirements for the F-35 programs was to minimize extra care needed for the RAM (Radar Absorbent Material). Unlike older stealth aircraft the F-35's ram is "baked in" to the aircraft skin, rather than being a coating. The F-117 and B-2 require climate controlled hangars because their coatings are old and delicate, the F-22 doesn't, but needs regular touch-ups for its coating, the F-35 is just left sitting outside most of the time regardless of where it's operating, a desert, the arctic, a jungle, the deck of a ship, you just leave it out there. The only common maintenance done on the F-35's RAM is replacing a relatively small amount of special RAM tape which is usually used around the edges of the access panels which are opened for other types of maintenance.

Cerebras has effectively 100% yield on these chips. They have an internal structure made by just repeating the same small modular units over and over again. This means they can just fuse off the broken bits without affecting overall function. It's not like it is with a CPU.

That's what it's running on. It's optimized for very high throughput using Cerebras' hardware which is uniquely capable of running LLMs at very, very high speeds.

You prevent enemy air defenses from shooting down your aircraft by blowing them up as part of SEAD/DEAD missions, which is exactly what the US did.

Takes much longer to build, requires a much larger up-front investment, and requires a lot more land.

The footprint needed when trying to generate this much power from solar or wind necessitates large-scale land acquisition plus the transmission infrastructure to get all that power to the actual data center, since you won't usually have enough land directly adjacent to it. That plus all the battery infrastructure makes it a non-starter for projects where short timescales are key.