1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage?
2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clusters, etc.
3. What's done to preserve order-of-operation within a kafka partition?
1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage?
2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clusters, etc.
3. What's done to preserve order-of-operation within a kafka partition?