mpconlen

You are right about the push/pull pattern, in the case the publisher crashes those jobs will be lost. That could be an issue depending on the use case and would be a great thing to address.

If I am understanding the other notes correctly:

Neither the worker nor publisher should ever really block (even in cases where M>N, N>M, publisher crashed, etc), mostly due to the async zmq bindings for node. If there are a ton of messages being sent to a single worker this worker will pull down messages as fast as it can and the rest will still be in publisher queue.

If a worker crashes this is happening separate from the publisher and won't affect the queue stored in the publisher's memory

I guess this should be clarified more in the README - this uses zmq (and potentially other brokers) as the queue, and uses the database for tracking results, saving recurring tasks, etc

These are both very good points.

The main reason I mentioned encrypting IRC instead of something like SILC is that many people already include irc in their daily workflow. Adding settings to a client you already use may be less intrusive than picking up a new protocol altogether.

The unfortunate reality seems to be that quality open source versions don't exist for a lot of these. I'd love to hear about alternatives that you know of though.