adrian17

(Note that this optimization can also trigger for plain classes (that’s LOAD_ATTR_INSTANCE_VALUE), it just needs a bit more guards and is only useful if you don’t abuse instances being dynamic objects. I just used slots as the most obvious example, but there are lots of different op specializations since like 3.11)

Fair, I don’t argue with that, I’m not knowledgeable about Java’s processes. Just one thing:

> The vast majority of JVM optimizations do not get a JEP.

In Python too. Still, the JIT really is just a really big thing. I don’t know if there could be a better way to do it, as either 1. You develop it off-branch and… make an enormous PR when you’re done, with risks of getting rejected anyway? Or 2, you allow more experiments on main like Rust, but then you end up with hundreds of minor half-maintained nightly-only features, possibly conflicting with each other, and eventually go through bureaucracy to remove them anyway.

> Optimizing for this would require a tree of hidden map objects much like JS JIT engines use.

I don’t really think that’s true or even that relevant , at least in context of classes (especially with slots - the class, by definition, _is_ the “hidden map”) and not generic dicts.

> The thing is, you can build that entire hidden type structure out, AND optimize against it, without ever having to mess with things like bytecode replacement.

Pretty sure the motivation was exactly the opposite - that a simple “if a LOAD_ATTR encounters the same class with slots N times in a row, swap it out for LOAD_ATTR_SLOT with inline cached slot index, and swap back to generic one if something changes” is way easier (both to implement and reason about) than any alternative that’d interact with the object model. (Also it means you skip having to branch between a slow and fast path at all)

> you can modify the property lookup algorithm in the VM itself (as in cpython c code) to use some annotation structure hanging off of the bytecode to look up inline caches of type-guarded fastpaths. It would be an optimized fastpath that integrated directly into the C code that interprets the GETATTR opcode.

Sorry, that sounds a bit vague and not too different from what CPython already is doing.

> It's from converting hashtable lookups into constant-time guard + offset lookups.

Python already does it in a lot of cases, see opcode specializations like LOAD_ATTR_INSTANCE_VALUE, LOAD_ATTR_SLOT.

Unfortunate. I've been following the work on it occasionally.

On one hand, it seemed like JIT work was really progressing in high gear until SC requested a new PEP. Also unfortunate, as it seemed like part of the hope for the JIT was that it'd counter the performance penalty from free threading.

On the other, it did feel (to me) like with all the fancy (and definitely cool) optimizations it got (automatic refcount operation removal, new type-based specializations, inplace math operations), the macrobenchmark benefits weren't _that_ exciting for a JIT. It definitely has potential for more, maybe with even fancier optimizations like unboxing ints, but the complexity just keeps going up.

> It said that the PEP was "overly optimistic" about user-visible differences with the JIT enabled.

The JIT is technically generated from the same source as the interpreter loop itself, so originally the differences were extremely unlikely, but I imagine the risk just kept going up as new JIT-only paths and optimizations were introduced.

And politically, I feel like SC’s reasoning was just “this side project has landed in main branch almost by accident and has been stewing without oversight ever since, with no stabilization in sight”; and the latest PEP was probably still too vague, with a lot of “here’s more things we can add to the stew” - so I can understand if SC wasn’t satisfied.

The big reddit thread on/r/cpp might actually be the _most_ prominent public example of how political it is (and how the committee operates in general), with several people involved with WG21 arguing in the open.

A random number I checked feels kinda optimistic, or there's something wrong with my setup; Gemma 26B-A4B on a 16GB 5060Ti is reported to get 21-88tok/s with Q4_K_M, meanwhile I can just barely eke out 70tok/s with Q3 (which fully fits in vram), and 40tok/s with Q4 (with recently-merged MoE cache on llama-cpp). I guess the average fits, but the peak feels implausible.

No clue about details, but Sebastian Raschka's site reports its kv cache usage at 210kB per token, which is like 6x more than this site claims ("0.273 GB per 8k"). Also, the site claims qwen 27b to use 2x more memory per token than gemma, which is the opposite from what I know.

Maybe I'm missing something, but why couldn't it be generated by the model? In older classification tasks with transformers like BERT, you could absolutely obtain a confidence score.

Agreed, clickbait. The only reason I clicked the article was because I _didn’t_ see any new ads after upgrading iOS, so I was curious what they meant.

It’s trading one property for another, isn’t it? Standard beziers with symmetric handles are C1 continuous but not necessarily G2 continuous; author’s scheme is the other way around, which makes them better for some use cases and worse for others.