Comment Comment by Lindon4290 Yeah, the rumours(?) are a groq system required to produce 300+ t/s on a Llama-2 70B (bs=1) requires 576 chips (9 racks) [1]So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs?[1] https://twitter.com/swyx/status/1759759125314146699 parent Lindon4290 2 years ago
So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs?
[1] https://twitter.com/swyx/status/1759759125314146699