There are three phases in auto-extraction...
- Acquisition: Greedy annotation of any possible data-parallel opportunities. This was originally done at the source-level when it was just 'xc'. For the C front end, it's been moved into the IR level, where the xc version will migrate to soon as well.
- Validation: Always done at the IR level, here we can check that there are no data-hazards, that a variable isn't being read from and written to in the same loop for example, or any aliased pointers, etc. etc. Anything that doesn't meet the cut has its annotation stripped, and we fall back to CPU, though it may still be vectorised on the CPU.
- Runtime measurement of which method is fastest, on the hardware that's actually running the code, with persistence to the local machines settings (defaults on a Mac, registry on Windows,...). If the routine passed muster during validation, it's guaranteed to be tile-able, and splittable over multiple GPUs in the system, or even networked co-operating machines (or Macs over Thunderbolt connections).
So, yes, the workers only see a tile of data, and there can't be a dependency within that tile, but xc allows trivial data-parallel annotation of loops. Here's a full-and-complete Mandelbrot that will run on a GPU:
#use "Stdio.xc"
#use "Par.xc"
#define W 2048
i32 main(void)
{
u32 total = 0;
par mandel :reduce(+ total)
{
for (u32 i in 0..W * W)
{
float cx = (float)(i % W) * (3.0f / W) - 2.0f;
float cy = (float)(i / W) * (3.0f / W) - 1.5f;
float x = 0.0f;
float y = 0.0f;
u32 k = 0;
while (k < 256 && x * x + y * y < 4.0f)
{
float xt = x * x - y * y + cx;
y = 2.0f * x * y + cy;
x = xt;
k = k + 1;
}
total = total + k;
}
}
printf("%u iterations\n", total);
return 0;
}
The system caches memory as it can:
- Discrete GPU: a GPU card (Vulkan or Cuda) keeps resident any array the program hasn't written since the last upload. Running a block again that uses that same data uploads nothing.
- Remote: a remote xcd keeps every slice and kernel it has been sent, so a repeat run sends hashes, not data. Same story as above for a discrete GPU managed via xcd
- Metal and integrated Vulkan: simd: memory isn't copied at all.
...so if you want something that uses a buffer from a previous data-parallel compute, have 2 'par' sections, and share the buffer. Each 'par' section will still run serially (en-masse) by default, but you ought to be able to ping-pong between a pair of them. I haven't yet run such a thing on the CFE :)
More specifically:
- Within a block, iterations must be independent. A loop where each iteration reads the one before (b[i] = b[i-1] + 3) stays an ordinary loop.
- Reductions are the one cross-tile dependency it handles. Each tile, on whatever device, produces a partial sum, min or max, and the runtime combines them. That's what the :reduce(+ total) does above.
- The scheduler sees the topology, not the workers. It splits a block into tiles and hands them out by each device's measured speed, local or remote. Each worker sees only its tiles plus the slices of input those tiles read.
- For streaming, two blocks in sequence works. Each block finishes on every device before the next statement runs, so the second can safely read the first's output. With CUDA residency, data the CPU hasn't touched in between isn't re-sent.
One thing I'm not trying to do is cover all possible cases automatically - that way lies unbounded complexity, and I'm a fan of simple code working well with low cognitive load. xc will work well automatically for non-self-dependent data, and allow you to work far more easily with self-dependent data.
- Acquisition: Greedy annotation of any possible data-parallel opportunities. This was originally done at the source-level when it was just 'xc'. For the C front end, it's been moved into the IR level, where the xc version will migrate to soon as well.
- Validation: Always done at the IR level, here we can check that there are no data-hazards, that a variable isn't being read from and written to in the same loop for example, or any aliased pointers, etc. etc. Anything that doesn't meet the cut has its annotation stripped, and we fall back to CPU, though it may still be vectorised on the CPU.
- Runtime measurement of which method is fastest, on the hardware that's actually running the code, with persistence to the local machines settings (defaults on a Mac, registry on Windows,...). If the routine passed muster during validation, it's guaranteed to be tile-able, and splittable over multiple GPUs in the system, or even networked co-operating machines (or Macs over Thunderbolt connections).
So, yes, the workers only see a tile of data, and there can't be a dependency within that tile, but xc allows trivial data-parallel annotation of loops. Here's a full-and-complete Mandelbrot that will run on a GPU:
The system caches memory as it can:- Discrete GPU: a GPU card (Vulkan or Cuda) keeps resident any array the program hasn't written since the last upload. Running a block again that uses that same data uploads nothing.
- Remote: a remote xcd keeps every slice and kernel it has been sent, so a repeat run sends hashes, not data. Same story as above for a discrete GPU managed via xcd
- Metal and integrated Vulkan: simd: memory isn't copied at all.
...so if you want something that uses a buffer from a previous data-parallel compute, have 2 'par' sections, and share the buffer. Each 'par' section will still run serially (en-masse) by default, but you ought to be able to ping-pong between a pair of them. I haven't yet run such a thing on the CFE :)
More specifically:
- Within a block, iterations must be independent. A loop where each iteration reads the one before (b[i] = b[i-1] + 3) stays an ordinary loop.
- Reductions are the one cross-tile dependency it handles. Each tile, on whatever device, produces a partial sum, min or max, and the runtime combines them. That's what the :reduce(+ total) does above.
- The scheduler sees the topology, not the workers. It splits a block into tiles and hands them out by each device's measured speed, local or remote. Each worker sees only its tiles plus the slices of input those tiles read.
- For streaming, two blocks in sequence works. Each block finishes on every device before the next statement runs, so the second can safely read the first's output. With CUDA residency, data the CPU hasn't touched in between isn't re-sent.
One thing I'm not trying to do is cover all possible cases automatically - that way lies unbounded complexity, and I'm a fan of simple code working well with low cognitive load. xc will work well automatically for non-self-dependent data, and allow you to work far more easily with self-dependent data.