While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy.
Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use.
Random combinations of tests also don't spark joy because this means both repetition (i.e. waste of resources) and testing potentially useless (unreachable or invalid) system states (both wastes resources and creates false alarms).
Ideally, the tests should be able to compose only in desired ways and rather than combining them randomly, there should be some deterministic process that creates a unique sequence or a tree of individual tests on subsequent runs. Ideally, such a test runner could also be configured to start with an existing system in a known state s.t. the tester can apply a patch and resume testing.
Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use.
Random combinations of tests also don't spark joy because this means both repetition (i.e. waste of resources) and testing potentially useless (unreachable or invalid) system states (both wastes resources and creates false alarms).
Ideally, the tests should be able to compose only in desired ways and rather than combining them randomly, there should be some deterministic process that creates a unique sequence or a tree of individual tests on subsequent runs. Ideally, such a test runner could also be configured to start with an existing system in a known state s.t. the tester can apply a patch and resume testing.