Ask HN: If kafka topic has X number partition then should I deploy X consumer?
Few documents say that consumer must equal to consumer count or unless it will slow down. I understand why it slows down.
So I did some simple math.
IF Kafka Throughput is 100K/s and takes 100ms to process it. It will need 10,000 partitions according to https://eventsizer.io/partitions .
Then should I deploy 10K consumers to process it without lag or latency? I think real-time chat app like `Discord` or `Slack` should have more throughput then 100K/s but I don't think they have 10K consumers up and running.
IF Kafka Throughput is 100K/s and takes 100ms to process it. It will need 10,000 partitions according to https://eventsizer.io/partitions .
Then should I deploy 10K consumers to process it without lag or latency? I think real-time chat app like `Discord` or `Slack` should have more throughput then 100K/s but I don't think they have 10K consumers up and running.
What am I missing here?