xgk

CPython could agree, and they have been asked to do so. But they refuse, because it would be a lot of work.

PyPy's problem with slow startup is, in principle, easily fixable.

When it was created, PyPy did not seek to optimise startup performance. Instead, its aim was steady-state performance of JITed code, and it succeeded handsomely. A conceptually simple fix would be to optimise traces in one processor core, while the interpreter runs in another. Once when the trace is finalised, does interpretation stop and switch to native execution. But PyPy is currently a sequential system and making it parallel, while conceptually easy, is a lot of work and needs a rewrite of the whole system. But, for better or worse, the PyPy team does not have the resources to do this.

The other of PyPy's big performance issues (Python code using a lot of FFI calls via a suboptimal FFI) is also, in principle, easily fixable. But this is a social problem: the Python community would have to move towards better FFIs.

In principle, it would also be easy to fix PyPy's resource problem. I am not sure about the social problem ...

I wonder if Claude does the brute-forcing directly. I think what's more likely is that Claude is synthesising programs that brute-force, albeit with interesting heuristics.

Mixture-of-Expert (MoE) was introduced in the 1990s [1, 2], see also [3, 4]. The idea was that MoE scales up model capacity and only introduces small computation overhead. MoEs did not become viable for high-performance applications until sparse routing was integrated with modern deep networks, made possible by large-scale distributed computation. The breakthrough came with the development of sparsely gated networks [5], which showed that it is possible to maintain model accuracy while activating only a small fraction of a large parameter network during both training and inference.

[1] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, Adaptive mixtures of local experts. (1991)

[2] M. I. Jordan, R. A. Jacobs, Hierarchical mixtures of experts and the EM algorithm. (1993)

[3] L. Xu, M. Jordan, G. E. Hinton, An alternative model for mixtures of experts. (1994)

[4] S. Waterhouse, D. MacKay, A. Robinson, Bayesian methods for mixtures of experts. (1995)

[5] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. (2017)

Indeed, that is the standard approach. It is also how some of the aforementioned languages desugar opaque type synonyms during compilation. It has the slight disadvantage that we can no longer use variables like

    x

in some situations, but need to use

    x._polynomials_gf_2

or whatever is the structure's field name. It is nice to avoid this boilerplate, which can become annoying quickly. Let the type-checker not the human do this work ...

> You do not need another language for this.

By the Church-Turing thesis you never need another language, but empirical practise has shown that the software engineering properties we see with real-world code and real-world programmers differ significantly between languages.

> CPUs actually implements 5 distinct data types

Yes, that's true, but the registers themselves are untyped, what modern CPUs really implement is multiple instruction semantics over the same bit-patterns. In short: same bits, five algebras! The algebras are given by different instructions (on the same bit patterns).

Here is an example, the bit pattern 1011:

• as a non-negative integer: 11. ISA operations: Arm UDIV, RISC-V DIVU, x86 DIV

• as an integer residue mod 16: the class [11] in Z/16Z. ISA operations: Arm ADD, RISC-V ADD/ADDI, x86 ADD

• as a bit string: bits 3, 1, and 0 are set. ISA operations: Arm EOR, RISC-V ANDI/ORI/XORI, x86 AND.

• as a binary polynomial: x^3 + x + 1. ISA operations: Arm PMULL, RISC-V clmul/clmulh/clmulr, x86 PCLMULQDQ

• as a binary polynomial residue modulo, say, x^4 + x + 1: the residue class of x^3 + x + 1 in GF(2)[x] / (x^4 + x + 1). ISA operations: Arm CRC32* / CRC32C*, x86 CRC32, RISC-V clmulr

And actually ... the floating point numbers also have the same bit patters, and could, in principle reside in the same registers. On modern ISAs, floats are usually implemented in a distinct register file.

You can use different functions in C on the bit patterns we call unsigned.

An alternative explanation might involve both of:

1. Stopping technology transfer worked exceedingly well for the west weakening the Soviet Union during the Cold War.

2. A distinct lack of (non-violent) alternatives for the West preventing China becoming the world's leading technological superpower (and hence also strongest military).

I doubt this will be successful in the long run, because China is not burdened by the Soviet Union's extremely inefficient way of organising its economy. Not to mention that China is the worlds biggest market.

Sorry, I should have said: ... (row)hammer hard enough, every sufficiently dense/modern DRAM ...

I think if you (row)hammer hard enough, every DRAM will eventually flip a bit.

Agree. The extreme secrecy of DRAM manufacturers about the innards of their chips puts an additional obstacles in the way of memory controllers (MCs) implementing efficient Rowhammer defences. In particular, if the MC doesn't know which addresses are corresponding to neighbouring rows, how can an MC know with certainty that any concrete row is being attacked? (And, to the best of my knowledge, DRAM manufacturers don't give away this information.)