saiojd

Plenty of people are already miserable. Might as well try if you are no?

What does __syncthreads() do here exactly? I'm new to CUDA, could get the overall idea of the FlashAttention paper but not the details.

I would guess, to make sure the classic noob mistake of typing `if x = 3:` instead of `if x == 3:` stays a syntax error

I had missed this message but have to say I love the expression "orthogonal nightmares", really captures it well :)

If this happens we will just get more things done with the same amount of compute (see: Blinn's law). The demand for GPUs does not really come from algorithmic compute requirements but from social expectation of progress in the field of AI. People will use all the compute they can get doing research using the budget they are given. What matters is how this budget is set.

Javascript's package management is so much better than Python's IMO...

Most papers already have code, and videos are very common.

I understand better what you mean now. Clearly you've thought about this topic more than I have, so I don't think I can contribute much to the conversation unfortunately.

"It still has to deliver value in return for the value it takes.": To make sure I understand, how do you define 'value' exactly here?

"makes the paid ads that do make it onto the platform strangely bolted on the side, not a smooth part of the experience.": I guess for me that's the appeal in Reddit, in that it is not completely "consumerfied" yet. I feel like a Wikipedia-type management would be a much better fit for the end-user, but obviously it would be harder to collect donations to run Reddit than to run Wikipedia.

"I agree that our broad culture sees little value in Reddit": I agree, that's really the crux of it and it's too bad.

I don't see your point, to be honest. This US government could easily afford to host Reddit. The reasons it does not are cultural, not economic.

That's a fair point about allowing competition. But frankly, it just sucks that Reddit choses to purposefully hurt its service. With social media, there's also a strong moating effect that is fundamentally anti-competitive.

I'm not the type to hate on capitalism but Reddit's management is truly capitalism gone wrong. Ruining a public service for the profits of a few. What a waste.

Thanks for the info. I didn't know about the TomeSD stuff, really interesting. Why do you think that AITemplate is so much faster?

I find you post intriguing. What would you say are the major janks with torch.compile, and what issues are addressed by TVM/AITemplate but not by torch.compile?

EDIT: If I understand correctly these libraries target deployment performance, while torch.compile is also/mostly for training performance?

Not to dismiss the guts of these explorers, but we have not been to the Moon in 50 years because it costs a fortune to get to when it's not much more than a floating rock (I'm being hyperbolic but the point is that there is no economic incentive to offset the cost). If we could get to it cheaply, but with a high risk of death, I'm sure many would be doing it right now.

`with` blocks in Python become very ugly when nested, I'm glad it's a single line personally