Well, one thing's for sure. Running Kafka on discrete VMs on a cloud provider "by the book" is ludicrously expensive.
I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant, AZ-redundant or globally redundant storage, most of which are available to run managed disks upon).
This is why properly designed Hadoop/Kafka cloud managed services employ storage adapters to leverage the provider's baked in redundancy. And why some cloud providers have Kafka-compatible event brokers.
The rest of what WarpStream does is just icing on the cake (although I'm curious as to the internals and how they avoid inter-AZ charges).
(full disclosure: I work at Microsoft, but built Hadoop/Spark/Kafka clusters before joining nearly a decade ago.)
I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant, AZ-redundant or globally redundant storage, most of which are available to run managed disks upon).
This is why properly designed Hadoop/Kafka cloud managed services employ storage adapters to leverage the provider's baked in redundancy. And why some cloud providers have Kafka-compatible event brokers.
The rest of what WarpStream does is just icing on the cake (although I'm curious as to the internals and how they avoid inter-AZ charges).
(full disclosure: I work at Microsoft, but built Hadoop/Spark/Kafka clusters before joining nearly a decade ago.)