Comment Comment by JPLeRouzic Only if you have access to corporate-level hardware:"It took us 48 hours to build the suffix array for RedPajama on a single node with 128 CPUs and 1TiB RAM" parent JPLeRouzic 1 year ago
"It took us 48 hours to build the suffix array for RedPajama on a single node with 128 CPUs and 1TiB RAM"