There are conduit actions (like dedup, sort) that require all the previous action's output to be ready and available before they can start producing any output. All the discussion of GC-friendliness of caches makes me think that there aren't any conduits that have more data than can fit in memory on a single machine.
If you had conduits that were too big to fit in memory at once, would you (channable) stream them to local disk (either explicitly or just using virtual memory)? Or would you be able to distribute work between multiple machines with a cluster-aware Conduit type?
Your scheduler could split the job up into multiple machines and run the same Conduit pipeline on all the machines, and only the conduit steps that need to communicate with each other would do so.
Separate question, do you have any actions that produce more output than their input? I could imagine some customers might find it useful to generate the cartesian product of two inputs, or the power set of one input.
If you had conduits that were too big to fit in memory at once, would you (channable) stream them to local disk (either explicitly or just using virtual memory)? Or would you be able to distribute work between multiple machines with a cluster-aware Conduit type?
Your scheduler could split the job up into multiple machines and run the same Conduit pipeline on all the machines, and only the conduit steps that need to communicate with each other would do so.
Separate question, do you have any actions that produce more output than their input? I could imagine some customers might find it useful to generate the cartesian product of two inputs, or the power set of one input.