Thank you for writing a clear explanation of how false positive rates determine your minimum change threshold. I've found it surprisingly difficult to explain this to developers/QA's without a basic statistical background.
What makes the situation worse, which you didn't mention, is that developers like to write a suite of benchmarks, each of which can result in a false positive regression. So even if the FP rate of an individual benchmark is <1%, you can easily end up at 10% FP rate if your suite is large enough.
What makes the situation worse, which you didn't mention, is that developers like to write a suite of benchmarks, each of which can result in a false positive regression. So even if the FP rate of an individual benchmark is <1%, you can easily end up at 10% FP rate if your suite is large enough.