> I would appreciate if you elaborated on separating step definitions from dependency definitions
As I said, I only very quickly skimmed since I'm busy, I might have overlooked information, and apologies in that case. But take the example from the front page:
evergreens.csv <- contracts.csv
grep Evergreen $INPUT > $OUTPUT
So now suppose a new requirement comes along - Evergreen is also called "Neverbrown" sometimes. It's decided the best way is to convert all references at input so nothing else gets confused downstream. So I need an extra step, now
renamed.csv <- contracts.csv:
sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT
evergreens.csv <- renamed.csv:
grep Evergreen $INPUT > $OUTPUT
Adding this step forced me to modify the declaration of the original command, even though what I added had nothing to do with that command. With Bpipe, for example, you say
extract_evergreens = {
exec "grep Evergreen $input > $output"
}
fix_names = {
exec "sed 's/Neverbrown/Evergreen/g' $input > $output"
}
Then you define your pipeline order separately -
run { fix_names + extract_evergreens }
If I get contracts from a different source that don't need the renaming, I can still run my old version and I'm not changing the definition of anything:
run { extract_evergreens }
Hope this explains what I mean, and again apologies if this is all clearly explained in your docs and I just jumped to conclusions from the simple examples!
As I said, I only very quickly skimmed since I'm busy, I might have overlooked information, and apologies in that case. But take the example from the front page:
So now suppose a new requirement comes along - Evergreen is also called "Neverbrown" sometimes. It's decided the best way is to convert all references at input so nothing else gets confused downstream. So I need an extra step, now Adding this step forced me to modify the declaration of the original command, even though what I added had nothing to do with that command. With Bpipe, for example, you say Then you define your pipeline order separately - If I get contracts from a different source that don't need the renaming, I can still run my old version and I'm not changing the definition of anything: Hope this explains what I mean, and again apologies if this is all clearly explained in your docs and I just jumped to conclusions from the simple examples!