taktoa

In section 4.4 it discusses the effect of the technique on Cadence Genus, which is a PD/synthesis tool. My point is that you have to flatten the graph at some point, and most of the benefit of flattening it later (keeping/making things vectorized) is to do higher level transformations, which are mostly not effective.

Regarding synthesis, I think approaches like this often seem promising to software engineers but ignore the realities of physical design. Hierarchical physical design tends to be worse than flat PD because there are many variables to optimize (placement density, congestion, IR drop, thermal, parasitics, signal integrity, di/dt, ...) and even if you have some solution in mind that optimizes area for a highly regular block, that layout could be worse than a solution that intersperses lower-power cells throughout that regular logic to reduce hotspots. And since placement is not going to be regular in any real design, delay won't be either, and there is a technique called resynthesis that restructures the logic network based on exactly which paths are critical which will essentially destroy whatever logic regularity existed.

The other thing is that high level optimizations tend to be hard to come by in hardware. Most datapath hardware is not highly fixed-function, instead it consists of somewhat general blocks that contain a few domain specific fused ops. So we either have hardware specifications that are natural language or RTL specifications that are too low level to do meaningful design exploration. Newer RTL languages and high level synthesis tools _also_ tend to be too low level for this kind of thing, it's a pretty challenging problem to design a formal specification language that is simultaneously high level enough and yet allows a compiler to do a good job of finding the optimal chip design. Approximate numerics are the most concrete example of this: there just aren't really any good algorithms for solving the problem of "what is the most efficient way to approximate this algorithm with N% precision", and that's not even including the flexibility vs efficiency tradeoff which requires something like human judgement, or the fact that in many domains it's hard to formulate an error metric that isn't either too conservative or too permissive.

I think what you're describing is SPMD, which is a compilation strategy, not a hardware architecture. I am not sure but I think SIMT is SIMD but with multiple program counters (1 per N lanes) to enable some limited control flow divergence between lane groups.

Rust and C++ implement generics with monomorphization rather than boxing, so there is a potential performance hit associated with a type like this in Java that is guaranteed not to exist in Rust.

In practice, the JVM may still monomorphize it, but it is not guaranteed to, and this would be a good reason to avoid unnecessary uses of generics in a high performance codebase like a kernel, if you chose to write one in Java.

Not just when the codomain is a field, but more generally when the codomain is itself a vector space. The former is a special case of the latter where you construct a 1D vector space from a field.

Definitely not an ARM situation.

> In a clocked design the clock signal needs to be routed to every element on the chip which requires a lot of power, the more so the higher the frequency is.

Clock only needs to be distributed to sequential components like flip flops or SRAMs. The number of clock distribution wire-millimeters in typical chip is dwarfed by the number of data wire-millimeters, and if a neural network is well trained and quantized activations should be random, so number of transitions per clock should be 0.5 (as opposed to 1 for clock wires), meaning that power can't be dominated by clock. The flops that prevent clock skew are a small % of area, so I don't think those can tip the scales either. On the other hand, in asynchronous digital logic you need to have valid bit calculation on every single piece of logic, which seems like a pretty huge overhead to me.

I git cloned the repo and then ran sloccount after checking out various commits (just did `git log | grep -C3 'Jan 1 [0-9:]* 2017'` or similar to find the relevant commits)

The open source release of XLA predates Lattner's tenure at Google by 7 months, and it definitely existed before that -- the codebase was already 66k SLOC at that point. During his tenure it went from 100k SLOC to 250k SLOC. It's now 700k SLOC. He also has, as far as I can tell, zero commits in the XLA codebase. "Of LLVM fame" would be more accurate I think.

I'm pretty sure those numbers are for training, not inference. I've run it on _CPU_ and gotten ~1 token per second.

Spanish flu is an orthomyxovirus, not a coronavirus. There are some cold-causing coronaviruses though.

The heat emitted by burning fossil fuels is completely irrelevant compared to the greenhouse impact. We burn about 11.7 gigatons of oil equivalent a year, which over 200 years would be 2.7 * 10^16 kWh. The greenhouse effect increase over the last 20 years leads to over 2.2 * 10^15 kWh/yr added to the planet, over 20x as much.

I feel like something important here that's been overlooked is how easy it is to get the driver into the socket. This places a very real limit on how fast a large number of screws can be driven. Torx has fairly small tolerances and can be annoying to get in; I've never used Robertson but it seems easier to get in.

Torx wood screws are common (albeit less common than Phillips, and more expensive) in US hardware stores

Monorepos are not mutually exclusive with binary dependencies.