Comment

Comment by Maro

We base our whole DS infrastructure on Airflow (and Superset):

http://bytepawn.com/fetchr-airflow.html

http://bytepawn.com/fetchr-data-science-infra.html

Airflow is somewhere between good enough and pretty cool, it's based on what we had at Facebook (called Dataswarm).

IMO in 2-3 years Airflow will be the de-facto ETL standard, like Hadoop used to be for "Big data". If you're rolling your own ETL at this point, you're wasting your time. If you're using something else, you're (probably) missing out on ETL-as-code goodness.

parent Maro 8 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton