Darn. Good catch. Have an upvote :)
I was all primed to give an explanation about some of the internals of the language, and after checking ... that's not whats going on here. What I thought was happening (and why I wasn't overly surprised by the saxpy results) was this:
- The above data is actually from my M4 Max MBP, not the PC using a 3090/5080/both
- xc has the concept of memory that is sufficiently aligned that it can be used directly with GPU data in a unified memory model. You just annotate the pointer with 'simd:', thus:
simd:u8* ptr = new u8[4096]
And what I had assumed was happening was that the C front-end was annotating its buffer correctly, that the kernel didn't need to do any buffer-copying (which is what generally slows down saxpy) and that the GPU was just ripping through the data in-place and handing back control to the CPU.
That ... turns out not to be the case [grin]. The CFE is not marking buffers that get allocated on the heap with the simd: annotation (this is a bug, I will fix it), so you are absolutely correct and the numbers as presented above don't stand up for saxpy. Here's a more in-depth table for saxpy, and it shows where the error crept in:
┌───────────────────────────────────┬───────────┐
│ saxpy, M4 Max (C, --par-auto) │ Time │
├───────────────────────────────────┼───────────┤
│ clang -O2, one thread │ 1,398 µs │
├───────────────────────────────────┼───────────┤
│ xcc, all CPU threads │ 518 µs │
├───────────────────────────────────┼───────────┤
│ xcc, auto (picks all CPU threads) │ 593 µs │
├───────────────────────────────────┼───────────┤
│ xcc, GPU │ 11,240 µs │
└───────────────────────────────────┴───────────┘
In fact the GPU-run's own report shows where the GPU time goes. Each saxpy pass takes about 0.8ms on the GPU itself, but just over 11ms end to end. Almost all of the difference is copying the two 64 MB arrays to and from the GPU-aligned buffer on every run. The 2.1× over clang is just memory bandwidth: one core can't saturate the M4 Max's memory bus and 16 can.
The 'auto' mode (the default unless you pin a kernel as must-use-this-execution-style) measures a few iterations of the work-tiles at runtime, and picks the fastest to run the full job with on the local hardware at runtime (unless it has metrics from a previous run).
Bottom line: what is actually happening here is that the 'auto' mode (also in the original table header) is doing its measurement, and picking the fastest route - which is the "all CPU threads" mode. You can see the measurement cost (I always run these tests after clearing out the persisted previous run, so there's always the measurement-cost in the 'auto' figure). What was misleading was the "(GPU)" in the original table header, I have to apologise for that.
Having said that, I'm glad you asked - in this case I don't think it'd make an actual difference (multi-threaded CPU still beats the non-buffer-transfer GPU) but potentially an integrated GPU (Mac or one of the "AI" Ryzens perhaps) might benefit from another, very slightly more complex compute, now that I know the automatic memory-alignment isn't working properly :)
I was all primed to give an explanation about some of the internals of the language, and after checking ... that's not whats going on here. What I thought was happening (and why I wasn't overly surprised by the saxpy results) was this:
- The above data is actually from my M4 Max MBP, not the PC using a 3090/5080/both
- xc has the concept of memory that is sufficiently aligned that it can be used directly with GPU data in a unified memory model. You just annotate the pointer with 'simd:', thus:
And what I had assumed was happening was that the C front-end was annotating its buffer correctly, that the kernel didn't need to do any buffer-copying (which is what generally slows down saxpy) and that the GPU was just ripping through the data in-place and handing back control to the CPU.That ... turns out not to be the case [grin]. The CFE is not marking buffers that get allocated on the heap with the simd: annotation (this is a bug, I will fix it), so you are absolutely correct and the numbers as presented above don't stand up for saxpy. Here's a more in-depth table for saxpy, and it shows where the error crept in:
In fact the GPU-run's own report shows where the GPU time goes. Each saxpy pass takes about 0.8ms on the GPU itself, but just over 11ms end to end. Almost all of the difference is copying the two 64 MB arrays to and from the GPU-aligned buffer on every run. The 2.1× over clang is just memory bandwidth: one core can't saturate the M4 Max's memory bus and 16 can.The 'auto' mode (the default unless you pin a kernel as must-use-this-execution-style) measures a few iterations of the work-tiles at runtime, and picks the fastest to run the full job with on the local hardware at runtime (unless it has metrics from a previous run).
Bottom line: what is actually happening here is that the 'auto' mode (also in the original table header) is doing its measurement, and picking the fastest route - which is the "all CPU threads" mode. You can see the measurement cost (I always run these tests after clearing out the persisted previous run, so there's always the measurement-cost in the 'auto' figure). What was misleading was the "(GPU)" in the original table header, I have to apologise for that.
Having said that, I'm glad you asked - in this case I don't think it'd make an actual difference (multi-threaded CPU still beats the non-buffer-transfer GPU) but potentially an integrated GPU (Mac or one of the "AI" Ryzens perhaps) might benefit from another, very slightly more complex compute, now that I know the automatic memory-alignment isn't working properly :)