aconz2

Ahh good to know about those -builder repos. It would be awesome to publish the compilers as container images, could also publish the squashfs as an artifact to a registry. And it shouldn't be too hard to back-publish the existing tarballs as containers without rebuilding. Might poke around a bit later

Thanks, will get that fixed

Yeah names are hard and it is just a working name. I'm simultaneously using containers as the distribution and runtime mechanism but also wanted them to be an implementation detail and not the spotlight

To add on since this took me a while to understand: for a single token, self attention is permutation invariant because we take the qK (one query dot all the other keys) weighted sum of all the values; that sum is what gives the invariance because + is commutative. But for all the tokens, the mha output matrix will not be invariant, but rather equivariant, where you apply the same permutation to the output matrix as you did to the input tokens. What might be a more useful example is to take one position, like the last one, and compute its mha for every permutation of the previous tokens; those will/should all be the same.

Great read and visuals. I think they typo'd the pun on basically/basisally. It got me thinking about program synthesis in the following scheme: data is embedded as vectors and program operations are metric tensors (or maybe just fields in general?) which tell the data how to move. Then, if you have an input/output pair we seek some program to move the data from input to output along some low energy path. Model a whole program as a time varying (t 0-1) metric tensor (is that a thing?) and optimize to find such an object. Maybe you choose ahead of time the number of operations you're searching over and these are like spline basis points and then you lerp between the metric tensors of each op; or you do it continuously and then somehow recover the operations. Then you want to find one program which satisfies multiple input/output pairs, ie one time varying metric tensor (or generally field) such that if you integrate from the input points they all end up at (or close to, which makes me think that you want some learned metric tensor for closeness) the output points. Right now I'm only thinking of unary ops with no constants, maybe the constants could be appended to the input data symbolically and you also get to optimize that portion of the input vectors, with the constraint that it is a shared parameter across all inputs.

surprised they didn't go straight into cloud-hypervisor, though I haven't actually tested with gpu yet but it is on my todo list. OCI layers can use zstd compression. I wonder if they are defeating layer sharing by splitting in 500 mb chunks. Lambda splits your image into chunks and shares at the block layer (I believe even same chunk different (user's?) container on a single host). Esp for 15 GB images I'd think using lazy pulling with nydus/stargz or whatever would be beneficial. I'd like to test out snapshotting, though my testing already boots a guest and runs a container in ~170ms; and I'm not actually sure how you write the guest init to signal it is ready for snapshotting and then wait properly (maybe you just sleep 1000?) so it resumes from the snapshot in a good state. I know fly has written about their use of snapshotting but I don't think it went into that detail. Cool stuff overall though, not worrying about locations and the yucky networking to do so seems nice

Bit late and I posted essentially this same comment on their site in hopes they'll see it, but this article jumped out at me because a while ago I was thinking about what it would take to get a whole organism single cell atlas so that you could explore it on the computer. The only thing I can really think of is to take a CNC machine, put it in a big freezer, then scan one layer of the specimen with a microscope and whatever multispectral lighting/imaging you need, then mill away a layer and repeat. Pathology slides are prepared with stains to enhance the contrast of membranes and other features which is probably not an option in the frozen state, so that's a big hurdle, but maybe you can do it with some fancy multispectral imaging. And if you can't, maybe you can at least start with lower resolution structures like anatomy (though I think there's still contrast challenges for that). I know there are some single cell atlas efforts for parts of or whole brains for mice and maybe other organs, but like the author's dream idea of getting order thousands of specimens to study variation, it would be great dataset if you could capture everything in one go.

Nice article, really like the survey bit. I know titles are just one tiny aspect, but they do set the tone, and in this case framing as a negative ("sad state") might put people off and may undermine the goal of getting people to adopt property-based testing. Though perhaps specifically this article is about stateful property-based testing, so it kinda comes off as negative against the existing library authors. I assume your intent is pure so rephrasing as "property-testing could be so much better" might be more positive. I am likely on the far end of sensitive on this only because I've spent a decade stuck in the negative view ("sad state" "everything sucks" "why did they design it that way" etc) and I'm trying to work my way out of it now. Anyways, too long rambling and mostly not even directed at the author necessarily but comment readers more generally.

As for how to get more adoption of property-based testing and/or stateful and/or parallel property-based testing, is there any way to do this from outside the language level? So you'd write your tests to consume commands and produce results from stdio, then an external process can drive the testing process. Can you run these in a VM to get deterministic thread scheduling for any language? Fakes could be implemented in any language and use the same stdio mechanism. And then build up a public set of known-good fakes. Ideally the fakes could be composable but not sure that is easy. Of course for bespoke things you'll need a bespoke fake, but I am thinking of the general stuff like a dictionary, queue, etc. This is all sparked because I've been thinking of how to build a giant inter-language inter-library testing database, initially with unit tests but this made me think about how to incorporate property based tests.

Late to the party, but I think my summary is (L is context length, C is hidden dimension, H is headsize, C = H * nh):

3.1 Optimised attention: Instead of using a learned W_V to project from C to H, slice V into H sized vectors. (V is just the input tokens X). This is because the matrix multiply is to a lower dimension anyway, so why not just slice. Slicing is just reshaping (L, C) -> (L, nh, H)

3.2 Efficient attention: I think this opens with a typo, "In the last section, we discussed how and why we can remove W_O..." should be W_V not W_O I think. Anyways, same as above, just for the keys this time. Reshape K (which is just X) from (L, C) -> (L, nh, H)

3.3 Super attention: Introduce an (L, L) W_A (lower triangular for masked) that transforms V on the left (X again) from (L, C) -> (L, C) (whereas standard attention has W_V (C, C) that transforms (L, C) -> (L, C) from the right). And they share W_A between heads.More efficient when C > L, so for long context models, probably not more efficient.

I think the first two modifications are equivalent to just setting W_V and W_K to the constant identity matrices right? So that makes me think what would happen if you instead restrict W_V (and/or W_K, W_Q) to be block diagonal (though non square) such that each head has in effect an (H, H) matrix which transforms the slice of X it receives. This is different than standard attention right? Because there the W_V acts over the full C dimension. Almost surely someone has thought of this so I will try to find out

Still learning so all this could be wrong