We have alerts set up that expect metrics for things like "orders placed" to always be happening at expected rates.
When datadog has the very rare outage that breaks ingestion, all of our alerts would normally go off because we aren't seeing the expected volume of "orders placed" and open up StatusPage incidents for us and our customers, call the pagers and get folks working.
But instead they automatically stop any false alerts that would normally alert here because of their outage. Saves me a lot of headaches.
It is stuff like this why I am happy paying the Datadog bills. Even their outages are good.
When datadog has the very rare outage that breaks ingestion, all of our alerts would normally go off because we aren't seeing the expected volume of "orders placed" and open up StatusPage incidents for us and our customers, call the pagers and get folks working.
But instead they automatically stop any false alerts that would normally alert here because of their outage. Saves me a lot of headaches.
It is stuff like this why I am happy paying the Datadog bills. Even their outages are good.