70b runs on 4x CS-3 estimated at $2-3m each, let’s say total system cost $10m, drawing ~100kW power. They don’t mention batch size, so let’s start with batch size 1 and see where we get. At 100% utilisation for 3 years that’d be 42 billion tokens for a cost of $10m capital plus ~$0.5m power and cooling let’s say, or $250 per million tokens. They’re claiming they can sell their API access at $0.60/million. To break even on this they’d need a batch size of 420. I don’t know how deep their pipeline is but Llama 3.1 70b has 80 layers with 6 meaningful matmuls per layer so it’s not a crazy multiple of that.
A single A100 processes at 13 t/s/u for batch 32. That costs $10k to process 39 billion tokens over 3 years = $0.25 tok/s/u. If you have batch size 420 you can do it even cheaper.
TL;DR: Cerebras are certainly advertising at a loss-leading price and will only have a viable product if they can get extraordinarily high utilisation of their system at this price. I don’t think they can, so they’re basically screwed selling tokens. Maybe this is to attract attention in the hope of selling hardware to someone willing to pay a premium for very low latency, but I suspect it’s just a means of getting one more round of funding in the hope of reducing costs in the next version.
A single A100 processes at 13 t/s/u for batch 32. That costs $10k to process 39 billion tokens over 3 years = $0.25 tok/s/u. If you have batch size 420 you can do it even cheaper.
TL;DR: Cerebras are certainly advertising at a loss-leading price and will only have a viable product if they can get extraordinarily high utilisation of their system at this price. I don’t think they can, so they’re basically screwed selling tokens. Maybe this is to attract attention in the hope of selling hardware to someone willing to pay a premium for very low latency, but I suspect it’s just a means of getting one more round of funding in the hope of reducing costs in the next version.