Sorry for what now appears to be a double-post. Engin had just registered for HN, hadn't seen his reply going through, so asked me to relay it.
Re-reading this Q+A this morning, I also wanted to clarify one thing, which is that when a 'foreach' or 'forall' does end up being executed on the CPU, that doesn't mean it has been serialized. 'foreach' loops on the CPU are candidates for vectorization while 'forall' loops typically result in multicore task-parallelism with each task also being a candidate for vectorization.
Re-reading this Q+A this morning, I also wanted to clarify one thing, which is that when a 'foreach' or 'forall' does end up being executed on the CPU, that doesn't mean it has been serialized. 'foreach' loops on the CPU are candidates for vectorization while 'forall' loops typically result in multicore task-parallelism with each task also being a candidate for vectorization.