fouronnes3

Back when Twitter was good there was an account called "Picard Management Tips" that shared such things and it was really golden.

What's stronger than Haskell?

I recently came across the existence of Penrose string diagram after curiously and naively researching the existence of index free notation for linear algebra. This seems to be a very interesting paper in the very same category of things I'd love to study and learn about but probably won't ever find the time!

Is a paper that publishes a 0.01% improvement of something at the cost of 5 times more power really an improvement? I believe that every single computer science measurement metric should have Joules or Watts in the denominator. If you are training a model I want to see performance per total energy consumed. If you are measuring inference accuracy, measure PER WATT.

I've always been a bit confused by the apparent tendency of the computer science field to mostly ignore energy and power. We are too often satisfied with the idea that software and programs exist in a perfect whiteboard world of xkcd 505 abstract compute.

I work on differentiable geometric optics with PyTorch. Seeing a list like this is really illustrative of the power that PyTorch provides when you start considering it like a general purpose GPU-enabled state of the art numerical optimization framework.

One thing I wonder is why no one has made a fork of PyTorch yet that removes all the API surface that doesn't produce GPU friendly code. Make dtype and device arg mandatory without defaults, remove in place operations that trigger a CPU sync, etc. This would increase confidence that written code will run on the GPU and pass torch.export() on the first try.

Thanks! Arbitrary precision arithmetic is definitely something I'd like to learn more about, yeah. Haven't had time to study it so much yet unfortunately.

I don't handle it, ahah. You are right that if you take any classical numerical computing algorithm and replace the floating point reals by interval unions, most of the time the number of intervals in the unions in each of your variables will grow very fast. This is one of the problems of unions and as far as I'm aware it's a topic of active academic research.

Yeah it's super interesting. Like you said, I learned that the IEEE 754 spec actually requires that complete implementations of floating point numbers expose a way to programmatically choose the rounding mode. As far as I know only C allows you to do that, and even then it depends on hardware support. For JS I had to use ugly typedarray casts. Which kinda only accidentally work due to endianess. But technically there should be an API for it!

There's other unused stuff in IEEE 754 like that: the inexact bit or signaling NaNs!

It's possible to support that but it makes the code very very much more complicated. I've decided early on to not support it. Would be a cool addition though!

I wonder if we could design a programming language specifically for teaching CS, and have a way to hard-exclude it from all LLM output. Kinda like anti virus software has special strings that are not viruses but trigger detections for testing.

This would probably require cooperation during model training, but now that I think of it, is there adversarial research on LLM? Can you design text data specifically to mess with LLM training? Like what is the 1MB of text data that if I insert it into the training set harms LLM training performance the most?

I've had a lot of enjoyment flipping the agentic workflow around: code manually and ask the agent for code review. Keeps my coding skills and knowledge of the codebase sharp, and catches bugs before I commit them!