This isn’t always a good way to test for bias in a model. Depending how the data is generated, and if the underlying population distribution isn’t taken into account then you may effectively be cherry picking results, and you could also run into the Yule-Simpson effect.
https://en.wikipedia.org/wiki/Simpson's_paradox
https://en.wikipedia.org/wiki/Cherry_picking