> who can't just now pluck messages off a queue to process
The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition.
If a certain message processing takes very long, all other messages in that partition will have to wait.
Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is much higher; but you do get nice message ordering for any given partition in kafka which you do not get in RabbitMq (afik); you are also get some really nice data locality with kafka because unless the consumers get the partitions re-assigned, all messages for the same key are served on the same physical consumer.
The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition.
If a certain message processing takes very long, all other messages in that partition will have to wait.
Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is much higher; but you do get nice message ordering for any given partition in kafka which you do not get in RabbitMq (afik); you are also get some really nice data locality with kafka because unless the consumers get the partitions re-assigned, all messages for the same key are served on the same physical consumer.