I write fast CUDA so I guess I'm one of the lucky few - which I already knew. I spent a lot of time benchmarking, instruction counting the PTX, function approximations, interleaving etc. I don't think it would cost that much money to write new ops for a new hardware for existing frameworks but for some reason very few do it - I figure it must be cheaper than making the hardware. I assume it's "penny wise, pound foolish" or maybe that it's very hard for hardware companies to do software as they try to pay their software guys like hardware guys.
More and more I'm doing more branchy algorithms so the CPU is now my new bottleneck - thank you fast GPUs - but I don't think the market will move to this way doing things before the current AI wave has crested. So it is probably not good business sense to target my use cases. I have quite a lot of concurrency so I think my ideal hardware is a whole lot of little CPU cores with decent cache and Half Matrix Multiply Accumulate (HMMA) instructions. Once my CPU bottleneck becomes too painful I'll look at what options are on the market. I think, but am not sure, the early Tesla AI chips focused too much on optimizing ResNet and then later they went with CPUs with the HMMAs. Instead of using a variety of instructions I can instead approximate most functions with more HMMAs.
EDIT: changed my reference of matmul intrinsics to the more precise HMMA instruction
More and more I'm doing more branchy algorithms so the CPU is now my new bottleneck - thank you fast GPUs - but I don't think the market will move to this way doing things before the current AI wave has crested. So it is probably not good business sense to target my use cases. I have quite a lot of concurrency so I think my ideal hardware is a whole lot of little CPU cores with decent cache and Half Matrix Multiply Accumulate (HMMA) instructions. Once my CPU bottleneck becomes too painful I'll look at what options are on the market. I think, but am not sure, the early Tesla AI chips focused too much on optimizing ResNet and then later they went with CPUs with the HMMAs. Instead of using a variety of instructions I can instead approximate most functions with more HMMAs.
EDIT: changed my reference of matmul intrinsics to the more precise HMMA instruction