> Well, when it comes to performance - you can’t use the average if you don’t know the distribution.
This is frankly wrong. Performance comes in multiple flavors. Latency is one of those, and there we know that percentiles really matter (see Andrew Certain's section of this talk: https://www.youtube.com/watch?v=sKRdemSirDM&feature=youtu.be... for Amazon's experience).
But for others, like throughput and scale, you don't need to know the distribution. In fact for throughput, the only thing that really matters is the long-term mean latency. For concurrency, it's that and long-term mean arrival rate. I wrote a blog post about it a while back (http://brooker.co.za/blog/2017/12/28/mean.html).
The core point here is that all summary statistics are misleading. You need to be clear on what you care about, and making absolute statements about the mean isn't a good way to do that.
Edit: This came across a bit more confrontational than I had intended. The OP makes some good points, but I think his point about the mean is overly broad.
This is frankly wrong. Performance comes in multiple flavors. Latency is one of those, and there we know that percentiles really matter (see Andrew Certain's section of this talk: https://www.youtube.com/watch?v=sKRdemSirDM&feature=youtu.be... for Amazon's experience).
But for others, like throughput and scale, you don't need to know the distribution. In fact for throughput, the only thing that really matters is the long-term mean latency. For concurrency, it's that and long-term mean arrival rate. I wrote a blog post about it a while back (http://brooker.co.za/blog/2017/12/28/mean.html).
The core point here is that all summary statistics are misleading. You need to be clear on what you care about, and making absolute statements about the mean isn't a good way to do that.
Edit: This came across a bit more confrontational than I had intended. The OP makes some good points, but I think his point about the mean is overly broad.