One of the problems described here seems to be that the people building the dashboards aren’t the ones adding the instrumentation. Admittedly I’ve only worked on one project that was all in on telemetry instead of using log analysis. And even that one had one foot in Splunk and one in Grafana, but I worked there long enough to see that we mostly only had telemetry for charts at least someone on call used regularly. I got most of them out of Splunk but that wasn’t that hard. We hadn’t bought enough horsepower from them that it didn’t jam up if too many people got involved in diagnosing production issues.
Occasionally I convinced them that certain charts were wrong and moved them to other stats to answer the same question, and some of those could go away.
I also wrote a little tool to extract all the stats from our group’s dashboard so we could compare used to generated and I cut I’d say about a third? Which is in line with his anecdote. I then gave it to OPs and announced it at my skip level’s staff meeting so other people could do the same.
Occasionally I convinced them that certain charts were wrong and moved them to other stats to answer the same question, and some of those could go away.
I also wrote a little tool to extract all the stats from our group’s dashboard so we could compare used to generated and I cut I’d say about a third? Which is in line with his anecdote. I then gave it to OPs and announced it at my skip level’s staff meeting so other people could do the same.