dimberman

In 2004, Mark Zuckerberg called us "dumb fucks" for giving him our addresses and photos. Now we're giving model providers our math theorems, pharmaceutical research, and startup business plans.

Hi @theptip,

I'm an Airflow PMC and would love to know a bit more about your comparison :).

1. Have you tried Airflow 2.0? We made some pretty big overhauls both in terms of UI and backend. 2. DAG versioning is currently problematic, but DAG versioning is a "when" and not an "if" so should be in a future 2.x version :). That said could you describe a bit more about your deployment issues? User stories like this help us improve the product. 3. Have you looked into using KEDA with the CeleryExecutor? You could create KEDA queues for a lot of commonly used workflows and then you'd only need to use the python or bash operator to run those tasks instead of k8spodop. 4. Are you using the Airflow helm chart or did you custom roll a deployment?

Any feedback would be highly appreciated and I'm also glad to answer any questions you might have!

I'd say it's simpler than that.

Airflow is a backend project built by backend engineers.

Most UI people don't use Airflow or know what it is.

@ryanhamilton is the first front-end dev to become a committer on the project and that JUST happened a few months ago.

We're hiring multiple UI experts to rebuild the Airflow UI from scratch in react, so this is just the beginning :).

Hi didip!

could you make an airflow issue related to that or start a thread in the dev list? That could be interesting! (though you might want to wait until after the holidays as we're all a bit wiped :) )

On the OSS side we have a helm chart that is heavily based on the one we use at Astronomer. That should hopefully get you started (or you can reach out to astro and someone will help out with a demo if you want help on that)

I'm sure they will eventually, but Astronomer and (I believe) Cloud Composer have day 1 support.

Airflow 2.0 :)

WAIT. Sorry I misspoke.

The only thing that is no longer allowed is using a bitshift operator between a DAG and a task.

  task_1 >> task_2

is totally fine

  my_dag >> task_1

Is no longer allowed. Most of your DAG should be completely fine. Apologies for the miscommunication.

I'm sure they will, I'm not sure on their timeline though.

So there are a few options for that if you're interested!

1. If you're using the KubernetesExecutor, you can point to custom images for individual tasks, this will primarily work if you're storing DAGs in git or a volume (or if you want to handle baking in DAGs for different images).

2. You can use custom images in KEDA queues. This way you can simply point to a queue for all tasks in that DAG and they will run in that environment.

3. You can use the k8spodoperator. Now that the k8spodoperator allows for templating, it would be pretty easy to create a template for a pod and just inject different commands for different steps.

Hope that helps!

Unfortunately that is one feature that we had to take out, but you should check out the TaskFlow API, it's a very worthwhile trade-off!

Edit:

Sorry I misspoke here

The only thing that is no longer allowed is using a bitshift operator between a DAG and a task.

  task_1 >> task_2

is totally fine

  my_dag >> task_1

Is no longer allowed. Apologies for the miscommunication.