The current solution in this space, that actually works really well at scale, is ElastAlert[0]. The problem is that ElastAlert is kind of a mess to work with. Lots of documentation, but you need to get into the weeds with it to figure out how it really functions.
Once you get it going it's a great tool. Scaling it out (we run hundreds of rules pretty frequently - upwards of 15 times an hour) is just standing up more instances with their own separate rules.
The current solution in this space, that actually works really well at scale, is ElastAlert[0]. The problem is that ElastAlert is kind of a mess to work with. Lots of documentation, but you need to get into the weeds with it to figure out how it really functions.
Once you get it going it's a great tool. Scaling it out (we run hundreds of rules pretty frequently - upwards of 15 times an hour) is just standing up more instances with their own separate rules.
[0] https://github.com/Yelp/elastalert