rnrn

Which model wrote this bizarre article?

Lots of emphasis on no kernels written and nothing recompiled but the intro and conclusion somehow suggest this is also the value of runinfra’s agents the rewrite software ?

Walk through the theory of peak decode throughout based on memory bandwidth with one weight load pass per token to provide a target for optimized decide but then doesn’t address low bandwidth utilization in that scenario at all and uses speculative decode where that theory doesn’t apply

looks like a bunch of boring desks for software developers, i don't see any electrical or computer engineers

> For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled.

This is not an accurate summary of what they developed.

Using .eh_frame to unwind stacks without frame pointers is not novel - it is exactly what it is for and perf has had an implementation doing it since ~2010. The problem is the kernel support for this was repeatedly rejected so the kernel samples kilobytes of stack and then userspace does the unwind

What they developed is an implementation of unwinding from an eBPF program running in the kernel using data from eh_frame.

No, wrong decade and wrong split - the test & measurement equipment and scientific equipment was long gone from HP at the time of the HP -> HP inc + HPE split. It ended up in Agilent (1999) and from there Keysight.

HP semiconductors went HP -> Agilent -> Avago, now broadcom.

How do get the weights for the right set of experts for a given batch of tokens into fast memory at the right time?

The activated experts is only available after routing, at which point you need the weights immediately and will have very poor performance if they are across PCIe

Why is there a new kernel driver here at all? It appears that all it does it allocate system RAM (“DDR4”) and export it as a dmabuf for import to cuda as mapped external memory. Then a userspace shim hijacks APIs to use that if gpu memory is full. cuda already supports allocating mapped system memory, so AFAICT this could be implemented in the userspace shim with no new kernel driver.

Also as other commenters have mentioned, redirecting allocations to managed memory would also enable similar oversubscription

And the hijack approach only makes sense for making apps have this behavior with no changes, and could be done with minor app changes (e.g. PyTorch has a pluggable allocator interface). App changes also enable intentionally placing specific allocations.

My impression is that this is vibe from beginning to end, starting from a design that only makes sense if you are hallucinating

This is not about reclaiming memory by swapping the contents out to disk. It is about killing processes due to having overcommitted beyond the available memory plus swap space. The processes thrown out of the plane (targeted by the OOM killer) cannot be resurrected

It’s not a cargo cult if the actions directly cause cargo to arrive based on well understood mechanics.

Regardless of whether it would be better in some situations to align to 128 bytes, 64 bytes really is the cache line size on all common x86 cpus and it is a good idea to avoid threads modifying the same cacheline.

> even on x86 on recent server CPUs, cache-coherency protocols may be operating at a different granularity than the cache line size. A typical case with new Intel server CPUs is operating at the granularity of 2 consecutive cache lines

I don’t think it is accurate that Intel CPUs use 2 cache lines / 128 bytes as the coherency protocol granule.

Yes, there can be additional destructive interference effects at that granularity, but that’s due to prefetching (of two cachelines with coherency managed independently) rather than having coherency operating on one 128 byte granule

AFAIK 64 bytes is still the correct granule for avoiding false sharing, with two cores modifying two consecutive cachelines having way less destructive interference than two cores modifying one cacheline.

Do you use some old version of iTunes to put music on it or are there other tools with better support for old iPods?

I thought this too, but after reading some of the other comments here I read some of the text of the chips act and the 2021 NDAA (mostly section 9902) and AFAICT Congress appropriated a bunch of money for financial assistance for semiconductor companies and gave the Dept of Commerce the authority to determine the funding type.

That they were grants instead of any other instrument appears to be a Biden Commerce decision, not a congressional one.

I’m no lawyer and could certainly be missing something i the law that says it has to be grants but from what I see it looks like figuring out what to do the the money was pretty much delegated to dept of commerce with limited direction about eligibility and review criteria.