I like to think (hope) that the next breakthrough will come not from these huge clusters, but from somebody tinkering with new ideas on a small local system.
I also wonder - is the compute the main limiting factor today ? Let's imagine there is an unlimited number of NVidia chips available right now and energy is cheap - would using a cluster x100 of current biggest one result in a significant improvement ? My naive intuition is that not really.
I also wonder - is the compute the main limiting factor today ? Let's imagine there is an unlimited number of NVidia chips available right now and energy is cheap - would using a cluster x100 of current biggest one result in a significant improvement ? My naive intuition is that not really.