mota7

"... and this is the whole ballgame: ", "jump out immediately", "... and the idea is disarmingly simple", etc etc.

Written by Claude AI. Which is a problem because Claude text is riddled with engagement bait and filler, making it exhausting to read.

Heat is mostly driven by leakage current and gate capacitance.

The big issue today is leakage currents. They typical account for around 30%-50% of total chip thermal budget, and they get increasingly difficult to control with smaller devices and lower voltages. They're also get worse with increased temperature(!).

The stacked devices here aren't the worst for leakage currents, but they're not fantastic either. Look at the 2nd graph in section 5: You'll see that the current never drops to zero over the range of gate-source voltages (for V_DS=0.7V). The minimum point is the best-case leakage current, and you can see it's well above zero! (The units on the vertical axis of the graph are unknown btw: The label reads as "current drain-source, arbitrary units")

If you have two threads on different cores that write to the same cacheline, the CPU has to enforce write ordering. The way it does this is for one of the cores to acquire a write lock on the cacheline. The actual implementation of this is via some variant of the MESI protocol that results in the cacheline on one core going to an Exclusive state, while copies of the cacheline on every other core becomes Invalid.

MESI takes a non-trivial amount of time to run. It's usually mediated at the L3 cache, which is frequently clocked slower than the main core, so it's a non-trivial number of cycles for a MESI state transition to happen (think at least ~40 cycles).

Compare that with a cacheline that's already Exclusive on a core: Writing to this cacheline is a pure L1 cache write at ~1 cycle, no MESI involved, no L3 cache involved.

Note that the typical userspace space mutex (some memory location that's modified by compare-exchange) is implicitly relying on MESI whenever two cores race for the lock, or whenever a lock is release on one core and then taken on another, etc etc.

So if your userspace coherency can be handled via some sharding that avoids needing MESI to run, it can go much faster than relying on the CPU's "internal mutex" aka MESI.

To directly address "is it possible that a hardware implementation of an algorithm could be slower than its software variant": The point is that with RSEQ, you can write a _different_ algorithm that's faster than an implementation relying on hardware mediated MESI. Instead of having memory that's accessed for a bunch of different cores, there's a chunk of memory per core, and each core almost always writes only to it's allocated chunk of memory.

Is there really that big a different in TFLOPS between the GB100 and GB202 chips? The GB100 has fewer SMs than the GB202, so I'm confused about where the 10x performance would be coming from?

Land the western australia wheat belt sells for less than $1000/acre. Is that very expensive?

Yes that bugged me too. If you replace 'precisely' with 'approximately' everywhere in the article it becomes much improved ;)

There's basically a difference in philosophy. GPU chips have a bunch of cores, each of which is semi-capable, whereas TPU chips have (effectively) one enormous core.

So GPUs have ~120 small systolic arrays, one per SM (aka, a tensorcore), plus passable off-chip bandwidth (aka 16 lines of PCI).

Where has TPUs have one honking big systolic array, plus large amounts of off-chip bandwidth.

This roughly translates to GPUs being better if you're doing a bunch of different small-ish things in parallel, but TPUs are better if you're doing lots of large matrix multiplies.

The hand-waving explanation: The slower you're going, the easier (cheaper) it is to change direction. And for eliptical orbits, the outer-most part of the orbit is where you're going slow.

So to make a drastic change in direction (aka, a very different orbit):

1. First burn to move far away from the center of the orbit so that you're going very slow.

2. Then burn to make a large change in direction (orbit).

3. Then wait until you cross your desired final orbit, and burn again to close it.

The tradeoff is that these types of orbit change are very slow (because you want to be going very slow for the middle burn, which means you take ages to get there).

Not quite: It's taking advantage of (1+a)(1+b) = 1 + a + b + ab. And where a and b are both small-ish, ab is really small and can just be ignored.

So it turns the (1+a)(1+b) into 1+a+b. Which is definitely not the same! But it turns out, machine guessing apparently doesn't care much about the difference.

The paper says "... optimized on next-word prediction only". Which is absolutely correct in 2023.

ChatGPT (and indeed all recent LLMs) using much more complex training methods than simply 'next-word prediction'.

Like most things, it's more complex than that, and as a result it can be either faster or slower than 'median(RTT to each DC in quorum)'.

It's a delicate balance based on the locations that rows are being read and written. In the case where a row being repeatedly written from only one location and not being read from different location, the writes can be significantly faster than would be naively expected.

> Gradient accumulation doesn't work with batch norms so you really need that memory.

Last I looked, very few SOTA models are trained with batch normalization. Most of the LLMs use layer norms which can be accumulated? (precisely because of the need to avoid the memory blowup).

Note also that batch normalization can be done in a memory efficient way: It just requires aggregating the batch statistics outside the gradient aggregation.

It's hard to convey just how ridiculously complex and expensive 5nm is compared with 90nm.

5nm is a multi-billion dollar fab, absurdly high running costs, and very expensive wafers.

As a data point: 5nm lithography equipment is > $100M/unit, needs megawatts of power to run, and they handle about 60-80 wafers per hour.

90nm lithography equipment is ~ $1M/unit, needs kilowatts of power to run, and they handle about 60-80 wafers per hour.

So a wafer scanned on 5nm equipment that could have been scanned on 9nm equipment is carrying a very large capital and operational cost burden that's pointless.