wanshao

Scaling Kafka clusters has always been a challenging task. Kafka uses the ISR multi-replica mechanism to ensure data persistence, which was a natural choice when the cloud was not yet mature. However, in 2024, when cloud computing is very mature, this design seems a bit outdated. When the cluster is scaled, we must move the data of the partitions. Moving partition data will affect normal reading and writing, and this process may last a long time, tens of minutes or a few hours, depending on the amount of your data. This means that often, even when the business scale has expanded to the critical point, we still dare not carry out such operations as expansion, because there is a high execution risk.

AutoMQ is fully aware of the root cause of this problem, so it has redesigned and implemented the entire storage layer of Kafka based on the cloud (we call ourselves cloud-first Kafka). Offload data persistence to cloud storage, and ensure data persistence by the multi-replica mechanism inside cloud storage. This also gives us the ability to build a more powerful self-balancing ability than Cruise Control. This blog post details how we technically achieve this, and we hope it can bring some new insights to everyone.