Since LinkedIn is using Kafka quite extensively, I'm wondering whether Azures poor support of Apache Kafka is at least partially responsible for this.
Azure is developing its own inhouse message broker that they call EventHubs. And while it is able to speak the Kafka protocol it has some weird limitations and quirks¹. I'm wondering why they are so hellbent on developing their own inferior version of something when they could just use the real deal and host it as SaaS.
¹ No support for KIP-71 (Enable log compaction and deletion to co-exist), limited to 10 topics (Azure lingo: EventHubs) per broker (Azure lingo: EventHubs namespace), mandatory client config parameters that you must not forget to set, even though the documentation just calls it "recommended" (https://learn.microsoft.com/en-us/azure/event-hubs/apache-ka...), some weird protocol level incompatibilities that are nowhere documented, for example a consumer group state transitions to "Dead" after the last consumer left instead of "Empty", and auto scaling that only ever scales up, but never down.
Azure is developing its own inhouse message broker that they call EventHubs. And while it is able to speak the Kafka protocol it has some weird limitations and quirks¹. I'm wondering why they are so hellbent on developing their own inferior version of something when they could just use the real deal and host it as SaaS.
¹ No support for KIP-71 (Enable log compaction and deletion to co-exist), limited to 10 topics (Azure lingo: EventHubs) per broker (Azure lingo: EventHubs namespace), mandatory client config parameters that you must not forget to set, even though the documentation just calls it "recommended" (https://learn.microsoft.com/en-us/azure/event-hubs/apache-ka...), some weird protocol level incompatibilities that are nowhere documented, for example a consumer group state transitions to "Dead" after the last consumer left instead of "Empty", and auto scaling that only ever scales up, but never down.