Author here. This is valid feedback, thank you. This post was originally an internal post. Internally, I have credibility as "the guy that does his homework". That doesn't carry over to the broader internet!
I think you're right that the data presented doesn't prove or disprove the thesis that "customer issues remain broadly stable". I think it lends credibility to the idea that "AI didn't lead to an obvious and immediate slopfest of bugs".
As for the bumps, we've noticed them annually and we haven't yet found a solid reason why. There’s no correlation with customer growth, PRs shipped, or features shipped. It's likely a combination of various seasonal effects in our own hiring and customer onboarding. What isn't in this article: we've more than doubled our customer base in this time period, our customers are more mature and raise bugs more readily, we've got ~60% more engineers, and our product surface area has grown a lot too. I'm yet to land on a satisfying metric that normalizes bug rates in a changing environment.
I think you're right that the data presented doesn't prove or disprove the thesis that "customer issues remain broadly stable". I think it lends credibility to the idea that "AI didn't lead to an obvious and immediate slopfest of bugs".
As for the bumps, we've noticed them annually and we haven't yet found a solid reason why. There’s no correlation with customer growth, PRs shipped, or features shipped. It's likely a combination of various seasonal effects in our own hiring and customer onboarding. What isn't in this article: we've more than doubled our customer base in this time period, our customers are more mature and raise bugs more readily, we've got ~60% more engineers, and our product surface area has grown a lot too. I'm yet to land on a satisfying metric that normalizes bug rates in a changing environment.