A workflow language is only as good as it’s engine.
Nextflow was mentioned. I think what most people want is probably closer to Airflow, although it takes some
time getting it up to production in a cloud environment (there is astronomer.io and a GCP product).
HTCondor via DAGMan has existed a long time, and there’s even engines built on that (Pegasus, Wings).
There’s Swift (http://swift-lang.org/main/) and it’s successor Parsl. Cray has Chapel. These are a bit different, in that they are more like a distributed computer program. Of course, so is Julia, but built into these languages is the assumption you can be using unreliable, in some way, computing. Makeflow and GNU Parallel are closer to this category too.
Then there’s Beam, but that’s dataflow.
The crappy thing about this is it’s hard to understand when to use a solution and when to not use a solution. Why are there so many solutions? Because there’s a ton of different needs, and a lot
of these focus on a few in particular:
Latency
Scalability or workers
Dynamic Scalability of workers
Throughput
Polyglot
Integration with existing Schedulers
Workflow Code Management (container support)
Maintainability of very large DAGs
Testability of DAGs/Development support
Execution Management support/Web APIs
Error recovery (especially for long running workflows)
Re-execution capabilities
Provenance tracking
Domain Specificity
Data Management (next to data processing)
... the list goes on.
Nextflow was mentioned. I think what most people want is probably closer to Airflow, although it takes some time getting it up to production in a cloud environment (there is astronomer.io and a GCP product).
HTCondor via DAGMan has existed a long time, and there’s even engines built on that (Pegasus, Wings).
There’s Swift (http://swift-lang.org/main/) and it’s successor Parsl. Cray has Chapel. These are a bit different, in that they are more like a distributed computer program. Of course, so is Julia, but built into these languages is the assumption you can be using unreliable, in some way, computing. Makeflow and GNU Parallel are closer to this category too.
Then there’s Beam, but that’s dataflow.
The crappy thing about this is it’s hard to understand when to use a solution and when to not use a solution. Why are there so many solutions? Because there’s a ton of different needs, and a lot of these focus on a few in particular:
Latency
Scalability or workers
Dynamic Scalability of workers
Throughput
Polyglot
Integration with existing Schedulers
Workflow Code Management (container support)
Maintainability of very large DAGs
Testability of DAGs/Development support
Execution Management support/Web APIs
Error recovery (especially for long running workflows)
Re-execution capabilities
Provenance tracking
Domain Specificity
Data Management (next to data processing)
... the list goes on.