Already well documented that there is huge variance in inference serving efficiency; at every layer (model, software, hardware) you have opportunities for gains. Similar to how “the Chinese” haven’t undercut on cloud providers’ CPU offerings, there’s a very good argument that the hyperscalers will manage to keep their vertically integrated hardware/software edge here too.
I don’t really know where this commodification idea came from, maybe MBA types that don’t actually understand how cloud economics work, applying basic first principles analysis?
The operational engineering that goes into serving a frontier model is _highly_ nontrivial, it is not commodifiable. This isn’t a SPA application.
I don’t really know where this commodification idea came from, maybe MBA types that don’t actually understand how cloud economics work, applying basic first principles analysis?
The operational engineering that goes into serving a frontier model is _highly_ nontrivial, it is not commodifiable. This isn’t a SPA application.