I just started using kSQL at work to build a data pipeline from Postgres to Elasticsearch. This will be replacing our error-prone batch processes. So far it has been a really nice experience, however I don't have it running in production yet.
Little bit concerned with how we will update the schema of the streams in production. At some point we will need to drop a stream and recreate it with a different schema, and I'm afraid we will miss some incoming data during the downtime. Confluent just released a migration tool, but I'm pretty sure this only works for adding/dropping columns and not changing the datatype of a column.
Little bit concerned with how we will update the schema of the streams in production. At some point we will need to drop a stream and recreate it with a different schema, and I'm afraid we will miss some incoming data during the downtime. Confluent just released a migration tool, but I'm pretty sure this only works for adding/dropping columns and not changing the datatype of a column.