In theory yes, you want a 1:1 partition to consumer ratio, that would give you the most optimal throughput, but it’s okay if you have less consumers than partitions, you’ll just have consumers doing more work.
Keep in mind you shouldn’t really go over 20k partitions in a Kafka cluster (recommended by Confluent), as that’s when things will start to get unstable. 10k is a lot of partitions, we have a hundred topics in our main cluster and only are at 15k partitions total. But if your running a high scale low latency environment then that sounds like what you’d need.
Keep in mind you shouldn’t really go over 20k partitions in a Kafka cluster (recommended by Confluent), as that’s when things will start to get unstable. 10k is a lot of partitions, we have a hundred topics in our main cluster and only are at 15k partitions total. But if your running a high scale low latency environment then that sounds like what you’d need.