What I like about ETL tools like Dagster and Prefect is that you get observability “for free”. You can set the granularity by deciding what is a task/job/flow/op and how they’re grouped together. And then in one UI you get logs, metrics, a waterfall view with timed executions, all kinds of useful information.
It’s so useful that sometimes I’m tempted to reach for it in non-ETL contexts. My problem is that these tools generally don’t mesh well with real-time streaming requirements.
It’s so useful that sometimes I’m tempted to reach for it in non-ETL contexts. My problem is that these tools generally don’t mesh well with real-time streaming requirements.