Heh, all the bioinformaticians come out of the woodwork :-)
Here's yet-another-project for bioinformatics workflows that I've been involved in. This one based on Groovy:
http://bpipe.org
I agree with your sentiments about the nature of pipelines vs build system a la make. Many many people start down the path of putting the classic DAG dependency analysis as the foundation of their needs when in fact, this isn't so much of a problem in real situations, and is even somewhat counterproductive because it forces you to declare a lot of things in a static way that actually aren't static at all. I've found tools like this completely break down when your data starts determining your workflow (eg: if the file is bigger than X I will break it in n parts and run them in parallel, otherwise I will continue on and do it using a different command entirely in memory).
In my experience the problems in big data analysis are more about the complexity of managing the process, achieving as much parallelization with as little effort and craziness as possible (don't see any mention of that in Drake), documenting what actually happened when something ran so you can figure it out later, and most of all, flexibility in modifying it since it changes every day of the week.
One mistake that Drake appears to make (again, from my quick skim), is interweaving the declaration of the "stages" of the pipeline (what they do) and the dependencies between them (the order they run in). This makes your pipeline stages less reusable and the pipeline harder to maintain. Bpipe completely separates these things out, which is something I like about it.
Here's yet-another-project for bioinformatics workflows that I've been involved in. This one based on Groovy:
http://bpipe.org
I agree with your sentiments about the nature of pipelines vs build system a la make. Many many people start down the path of putting the classic DAG dependency analysis as the foundation of their needs when in fact, this isn't so much of a problem in real situations, and is even somewhat counterproductive because it forces you to declare a lot of things in a static way that actually aren't static at all. I've found tools like this completely break down when your data starts determining your workflow (eg: if the file is bigger than X I will break it in n parts and run them in parallel, otherwise I will continue on and do it using a different command entirely in memory).
In my experience the problems in big data analysis are more about the complexity of managing the process, achieving as much parallelization with as little effort and craziness as possible (don't see any mention of that in Drake), documenting what actually happened when something ran so you can figure it out later, and most of all, flexibility in modifying it since it changes every day of the week.
One mistake that Drake appears to make (again, from my quick skim), is interweaving the declaration of the "stages" of the pipeline (what they do) and the dependencies between them (the order they run in). This makes your pipeline stages less reusable and the pipeline harder to maintain. Bpipe completely separates these things out, which is something I like about it.