Comment

Comment by gtrubetskoy

If you are using BigQuery and your "workflow" amounts to importing data from Postgres/MySQL databases into BQ and then running a series of SQL statements into other BigQuery tables - you might want to look at Maestro, it written in Go and is SQL-centric, there is no Python dependency hell to sort out:

https://github.com/voxmedia/maestro/

With the SQL-centric approach you do not need to specify a DAG because it can be inferred automatically, all you do is maintain your SQL and Maestro takes care of executing it in correct order.

parent gtrubetskoy 6 years ago

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton