At my last job (prior to joining elasticsearch), I had a cluster of 7 machines (16 cores, 16gb ram, 2TB raid1), each running logstash and elasticsearch.
The event rate going into this cluster was about 5000 events/sec on average (burst up to 10,000 events/sec sometimes).
During a maintenance (two machines going offline for disk repairs), I benchmarked the surviving 5-node cluster at 88,000 events/sec peak performance.
In terms of capacity planning, this means that we could have a 9x increase in normal event load and still not need to grow the cluster's processing capacity.
Persistent storage is another story. We stored about 300GB/day of events, getting us roughly 45 days of data retention before we would run out of space (2TB * 7 nodes / 300gb/day; roughly 45 days). I'm working on improving storage efficiency of logstash and elasticsearch, too, so retention should improve greatly in the long term.
For other experiences, it's useful to invoke the community and ask what others are done - the #logstash irc channel on freenode is very active as is the [email protected] mailling list.
Hope this helps!
The event rate going into this cluster was about 5000 events/sec on average (burst up to 10,000 events/sec sometimes).
During a maintenance (two machines going offline for disk repairs), I benchmarked the surviving 5-node cluster at 88,000 events/sec peak performance.
In terms of capacity planning, this means that we could have a 9x increase in normal event load and still not need to grow the cluster's processing capacity.
Persistent storage is another story. We stored about 300GB/day of events, getting us roughly 45 days of data retention before we would run out of space (2TB * 7 nodes / 300gb/day; roughly 45 days). I'm working on improving storage efficiency of logstash and elasticsearch, too, so retention should improve greatly in the long term.
For other experiences, it's useful to invoke the community and ask what others are done - the #logstash irc channel on freenode is very active as is the [email protected] mailling list.
Hope this helps!