Not a lot of public data about things like that at scale, but there are some things like the Meta Llama report that had ~ 9% annualized gpu failure rate while training Llama 3. Not 100% clear from the Meta report if those are dead hardware or just 'unhappy' because they just put them into the 'interruption caused by' bucket.
Would be interested if anyone had better data on actual live failure rates of h100 or better class gpu that suck so much power.
Not a lot of public data about things like that at scale, but there are some things like the Meta Llama report that had ~ 9% annualized gpu failure rate while training Llama 3. Not 100% clear from the Meta report if those are dead hardware or just 'unhappy' because they just put them into the 'interruption caused by' bucket.
Would be interested if anyone had better data on actual live failure rates of h100 or better class gpu that suck so much power.
https://ai.meta.com/research/publications/the-llama-3-herd-o... https://www.trendforce.com/news/2024/10/31/news-datacenter-g...