> Compute has been outpacing memory for decades. Like CPUs, GPUs have countered this with increasingly sophisticated caching strategies.
I'd say it's rather the contrary. Unlike CPUs, GPUs don't attempt to directly counter this. By accepting higher latencies, they have let themselves parallelize much more widely (or wildly) relative to CPUs - and the high number of parallel pseudo-threads provides a "latency hiding" effect.
This effect is illustrated for example, in this presentation on optimizing GPU code:
I'd say it's rather the contrary. Unlike CPUs, GPUs don't attempt to directly counter this. By accepting higher latencies, they have let themselves parallelize much more widely (or wildly) relative to CPUs - and the high number of parallel pseudo-threads provides a "latency hiding" effect.
This effect is illustrated for example, in this presentation on optimizing GPU code:
https://www.olcf.ornl.gov/wp-content/uploads/2019/12/03-CUDA...
(lame) animation on slide 11 and onwards.