What I would expect a system like this to have, at a minimum:
* Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions)
* Multiple stopping points depending on said statistics.
* Automatic isolation to the greatest extent possible (given appropriate permissions)
* Interleaved execution, in case something external changes mid-way.
I don't see any of this in hyperfine. It just… runs things N times and then does a naïve average/min/max? At that rate, one could just as well use a shell script and eyeball the results.
Is it, though?
What I would expect a system like this to have, at a minimum:
I don't see any of this in hyperfine. It just… runs things N times and then does a naïve average/min/max? At that rate, one could just as well use a shell script and eyeball the results.