Cool, this definitely seems like a good enumeration of techniques, nice to see that they discuss stuff like kernel
fission as well. Having a good understanding of loop nest optimization transformations (tiling, fission, fusion, strip mining/sinking, iteration order changes, etc) provides a good vocabulary for talking about this stuff too.
As someone who has spent 80%+ of their time CUDA programming for the past 9 years (I wrote the original GPU PyTorch tensor library, the Faiss GPU library, and several things that Nvidia took and put into cuDNN), I found the most instructive, short yet "advanced" education on the subject to be Paulius Micikevicius' various slide decks on "Peformance Optimization"; e.g.:
https://on-demand.gputechconf.com/gtc/2013/presentations/S34...
(there are some other ones outstanding, think one was for the Volta architecture as well)
They're old but still very relevant to today's GPUs.
As someone who has spent 80%+ of their time CUDA programming for the past 9 years (I wrote the original GPU PyTorch tensor library, the Faiss GPU library, and several things that Nvidia took and put into cuDNN), I found the most instructive, short yet "advanced" education on the subject to be Paulius Micikevicius' various slide decks on "Peformance Optimization"; e.g.:
https://on-demand.gputechconf.com/gtc/2013/presentations/S34...
(there are some other ones outstanding, think one was for the Volta architecture as well)
They're old but still very relevant to today's GPUs.