The system I built was called exacycle. It ran as a borg job in every cluster we had access to, with priority 0 or 1, with N tasks (where N=# of machines in the cluster). Borg ran the "worker" task as long as there were resources free on the machine (often, several workers ran on the same machine, and no workers ran on other machines). There was a global schedulers that picked mini-tasks off a global queue and fed them to the cluster-level scheduler (batches based on instant resource availability, and subject to some data locality optimization) which then fed them to the machine workers.
At peak, we had 1-3 million xeon cores running our tasks (protein folding, protein design, drug docking), 24/7. This was before GPUs were really a thing. Each mini-task was basically a rosetta or gromacs or autodock binary running on local inputs and writing local outputs for ~1 hour using ~1GB RAM and no network. Input and output was via colossus, or distributed through a global torrent network (literally all 1-3 million cores formed a single data transfer group, with data passed between clusters at low probability).
I built most of the code of the system (which depended on Google infra) including the scheduler and the machine worker, and also ran the system as on-call "SRE" for 3 years. During that time, we nearly almost didn't take down the Google global network, and had small latency impacts on many services. Managing to run the system for so long without antagonizing ads or search or youtube was a major challenge.
I would say the complexity of the system I built (considering everything down to the OS) was roughly the level of a few human cells in a plate, in terms of interacting components and the complexity of their interactions.
Here are some of the publications that came from running exacycle (we provided strong evidence that using more compute improves outcomes in molecular simulations)
https://pubmed.ncbi.nlm.nih.gov/24345941/
https://pubmed.ncbi.nlm.nih.gov/24265211/
At peak, we had 1-3 million xeon cores running our tasks (protein folding, protein design, drug docking), 24/7. This was before GPUs were really a thing. Each mini-task was basically a rosetta or gromacs or autodock binary running on local inputs and writing local outputs for ~1 hour using ~1GB RAM and no network. Input and output was via colossus, or distributed through a global torrent network (literally all 1-3 million cores formed a single data transfer group, with data passed between clusters at low probability).
I built most of the code of the system (which depended on Google infra) including the scheduler and the machine worker, and also ran the system as on-call "SRE" for 3 years. During that time, we nearly almost didn't take down the Google global network, and had small latency impacts on many services. Managing to run the system for so long without antagonizing ads or search or youtube was a major challenge.
I would say the complexity of the system I built (considering everything down to the OS) was roughly the level of a few human cells in a plate, in terms of interacting components and the complexity of their interactions.
Here are some of the publications that came from running exacycle (we provided strong evidence that using more compute improves outcomes in molecular simulations)
https://pubmed.ncbi.nlm.nih.gov/24345941/ https://pubmed.ncbi.nlm.nih.gov/24265211/