Airflow helped my team out a lot a couple years ago mainly for the simplicity of the topdown UI-based view of a complicated ETL AND the ability to retry parts of the ETL.
We had lots of lessons learned. For instance, why does PythonOperator even exist? It takes a callable and thus you're likely not going to see good coding pattern emerge for something that needs to be 1000+ LoC. Instead, we just subclassed BaseOperator and used tried-and-true OO principles.
We had lots of lessons learned. For instance, why does PythonOperator even exist? It takes a callable and thus you're likely not going to see good coding pattern emerge for something that needs to be 1000+ LoC. Instead, we just subclassed BaseOperator and used tried-and-true OO principles.