Luigi is a great Python library for building multi-stage pipelines with parameterized tasks. It easily extend to new storage types and targets.
We currently use it to build a moderately complex product from dozens of data streams (files, APIs and things in-between) and millions of records. At its core is a DAG and topological sort, which capture the essence of pipelines and execution.
A DAG is easy to visualize, even its development over time[1].
We currently use it to build a moderately complex product from dozens of data streams (files, APIs and things in-between) and millions of records. At its core is a DAG and topological sort, which capture the essence of pipelines and execution.
A DAG is easy to visualize, even its development over time[1].
[1] https://imgur.com/a/RuC0B5Y (generated from the projects' code, in this case https://github.com/miku/siskin)