Hi all, Logstash developer here. It's always exciting to see new stuff in this space, however, this post has me confused. Maybe the OP can clue me in.
I'm a bit confused as the assertion "This worked for a while, but when we wanted to make our pipeline more fault-tolerant, Logstash required us to run multiple processes.", is no more true for Logstash than it is for any other piece of software. Single processes can fail, so it can be nice to run multiples. It would be great if the author of the piece had clarified that further. If you're around I'd love to hear specifically what you mean by this. Internally Logstash is very thread friendly, we only recommend multiple processes when you want either greater isolation or greater fault tolerance.
I don't personally see what the difference is between:
Filebeat -> LogZoom -> Redis -> Logstash -> (Backends)
and
Filebeat -> LogStash -> Redis -> Logstash -> (Backends)
or even better
Filebeat -> Redis -> Logstash -> (Backends)
You can read more about the filebeat Redis output here: https://www.elastic.co/guide/en/beats/filebeat/current/redis...
I'm a bit confused as the assertion "This worked for a while, but when we wanted to make our pipeline more fault-tolerant, Logstash required us to run multiple processes.", is no more true for Logstash than it is for any other piece of software. Single processes can fail, so it can be nice to run multiples. It would be great if the author of the piece had clarified that further. If you're around I'd love to hear specifically what you mean by this. Internally Logstash is very thread friendly, we only recommend multiple processes when you want either greater isolation or greater fault tolerance.
I don't personally see what the difference is between:
Filebeat -> LogZoom -> Redis -> Logstash -> (Backends)
and
Filebeat -> LogStash -> Redis -> Logstash -> (Backends)
or even better
Filebeat -> Redis -> Logstash -> (Backends)
You can read more about the filebeat Redis output here: https://www.elastic.co/guide/en/beats/filebeat/current/redis...