vlejd

This is awesome. I have been trying to track similar things mainly to identify brands of cloths that actually last long enough to be worth it.

How do you keep the discipline to do this?

I also assume it is very nice to have such list if something gets broken/torn/lost and you need to buy a replacement right aways.

How much testing did it took Arive to such stable set?

Interestingly enough, we found that cublas is not that well optimized for some less common consumer GPUs, specifically 3090. We saw that is didn't really achieve it's full potential for a lot of different matrix shapes, probably because of poor tuning. Interestingly enough, out kernel does not have any parameters, and it was able to outperform cublas even in setting where it has no right to do so.

Regarding patterns, we tested mainly random matrices and ones created by Wanda pruning. 2:4 sparsity (commonly used structure) will have same results as random matrix (probably even better). Interestingly enough, block sparsity could have very close to a worst case scenario with our format, because it promotes disproportional long sequences of zeroes.

Regarding other usecases, we are looking into it, but most common ones we found are usually for much smaller sparsity <1%. If you know about some other use case that is in the 30-90 range, let us know.

I think the lack of efficient GPU kernels was the main problem. It is much, much easier to get a real speedup and memory reduction from quantization from fp16 to fp8 than from 50% sparsity. For sparsity you needed structure (which makes your model worse) and special hardware support.

Internet is dead indeed. Amazing idea! Will use it to test my posts.

Thant looks very nice. I am writing some technical blogs from time to time and this will come in handy. How does it work with the code execution though? Are you running some service somewhere that takes care of it, or is it in browser?