> Weirdly, underutilization of hardware was never a problem before.
Fixing exactly this has been the constant drumbeat of IT since the early 00's at least. It started with virtualization, to get capacity factor of hardware up in the enterprise. The average server back before VMWare (and HyperV/etc.) became a major thing was probably sitting at 10% utilization or less 95% of the time. Hypervisors were invented to fix this as their primary use case. You'd take a 5-10% performance hit on the raw server performance to gain 90% efficiency back.
Next came the cloud. Which was virtualization at scale. The larger the scale, the better the efficiency.
It's basically almost all anyone has actually accomplished in IT over the past decade or two. Everything else has been largely stagnant compared to the 80's and 90's.
AI inference is in it's infancy, but I imagine we will see more or less the exact same thing play out yet again. Lots of initial on-prem stuff due to cost and freedom, and eventually it becomes virtualized, commoditized, customer segregated, etc. Just like average cloud workloads are today and all the various "small" megawatt-scale builds moved back into the hyperscaler type providers within the next 5-10 years.
> How many companies are building datacenter commons and sharing with other companies to optimize resource usage?
Quite a few that I've seen. Rented inference/token factories are the current new hotness. Plenty of speculative builds being done by hundreds of smalltime providers. These will consolidate per usual and everyone but the last few top percent will remain as the winners. Just like the first hosting providers. Virtualization and fancy schedulers to maximize hardware utilization across a number of end-user customers are a huge topic in this space at the moment. Heck, a lot of these small time players are providing raw compute for the giant AI labs and even hyperscalers.
I don't like that Altman is right on this btw. I much preferred the highly inefficient days of every midsize corporate office having a 200-500kw datacenter in the basement. AWS and their ilk are quite obviously orders of magnitude more hardware efficient though.
Fixing exactly this has been the constant drumbeat of IT since the early 00's at least. It started with virtualization, to get capacity factor of hardware up in the enterprise. The average server back before VMWare (and HyperV/etc.) became a major thing was probably sitting at 10% utilization or less 95% of the time. Hypervisors were invented to fix this as their primary use case. You'd take a 5-10% performance hit on the raw server performance to gain 90% efficiency back.
Next came the cloud. Which was virtualization at scale. The larger the scale, the better the efficiency.
It's basically almost all anyone has actually accomplished in IT over the past decade or two. Everything else has been largely stagnant compared to the 80's and 90's.
AI inference is in it's infancy, but I imagine we will see more or less the exact same thing play out yet again. Lots of initial on-prem stuff due to cost and freedom, and eventually it becomes virtualized, commoditized, customer segregated, etc. Just like average cloud workloads are today and all the various "small" megawatt-scale builds moved back into the hyperscaler type providers within the next 5-10 years.
> How many companies are building datacenter commons and sharing with other companies to optimize resource usage?
Quite a few that I've seen. Rented inference/token factories are the current new hotness. Plenty of speculative builds being done by hundreds of smalltime providers. These will consolidate per usual and everyone but the last few top percent will remain as the winners. Just like the first hosting providers. Virtualization and fancy schedulers to maximize hardware utilization across a number of end-user customers are a huge topic in this space at the moment. Heck, a lot of these small time players are providing raw compute for the giant AI labs and even hyperscalers.
I don't like that Altman is right on this btw. I much preferred the highly inefficient days of every midsize corporate office having a 200-500kw datacenter in the basement. AWS and their ilk are quite obviously orders of magnitude more hardware efficient though.