> A histogram of the duration it took to serve a response to a request, also labelled by successes or errors.
I recommend against this, rather have one overall duration metric and another metric tracking a count of failures.
The reason for this is that very often just the success latency will end up being graphed, and high overall latency due to timing-out failed requests will be missed.
The more information you put on a dashboard, the more chance someone will miss a subtlety like this in the interpretation. Particularly if debugging distributed systems isn't their forte, or they've been woken up in the middle of the night by a page.
This guide only covers what I'd consider online serving systems, I'd suggest a look at the Prometheus instrumentation guidelines on what sort of things to monitor for other types of systems: https://prometheus.io/docs/practices/instrumentation/
I recommend against this, rather have one overall duration metric and another metric tracking a count of failures.
The reason for this is that very often just the success latency will end up being graphed, and high overall latency due to timing-out failed requests will be missed.
The more information you put on a dashboard, the more chance someone will miss a subtlety like this in the interpretation. Particularly if debugging distributed systems isn't their forte, or they've been woken up in the middle of the night by a page.
This guide only covers what I'd consider online serving systems, I'd suggest a look at the Prometheus instrumentation guidelines on what sort of things to monitor for other types of systems: https://prometheus.io/docs/practices/instrumentation/