Comment by akssri

The benchmark is flawed. Cupy follows Numpy in doing a type-promotion of the data to float64 before computing the covariance.

https://github.com/cupy/cupy/blob/b19577199d3f75c76e23c75194...

The following function in PyTorch completes the float32 computation in 27 ms (1080 Ti). I'm fairly sure that a similar function in Cupy would achieve very similar results.

  def corr(dat):
      cov = (dat @ dat.T).div_(dat.shape[-1]-1)
      sigma = torch.diag(cov).sqrt_()    
      return cov.div_(sigma[None]).div_(sigma[None].T)

The heart of the computation SGEMM is in anycase provided by CUBLAS, and the only areas in which PyT and Cupy differ are in the tricks used to unroll the loop for the broadcasting & L1 op. The performance difference b/w PyT, Cupy and others should be negligible IMO.

Note: np.corrcoef takes about 1.12 s and the above float32 version takes 325ms on the CPU.