My experience is that this is difficult without the system under test being designed to operate deterministically.
So for example: the system clock needs to be replaceable with a fake, or the current time needs to be injected, or perhaps both.
Another example: Go channels. These are a real PITA to surface in a way that allows a test harness to drive them deterministically.
A while ago I built a simulator testing harness for an early version of the Knative autoscaler[0]. I eventually ran aground on the difficulty of keeping up with changes, but it may be rearchitected and repurposed to assist Kubernetes HPA development instead.
FoundationDB's approach is to pretty much bake optional determinism into the language with some macro magic. It makes me wish that were easier to do in other languages.
So for example: the system clock needs to be replaceable with a fake, or the current time needs to be injected, or perhaps both.
Another example: Go channels. These are a real PITA to surface in a way that allows a test harness to drive them deterministically.
A while ago I built a simulator testing harness for an early version of the Knative autoscaler[0]. I eventually ran aground on the difficulty of keeping up with changes, but it may be rearchitected and repurposed to assist Kubernetes HPA development instead.
FoundationDB's approach is to pretty much bake optional determinism into the language with some macro magic. It makes me wish that were easier to do in other languages.
[0] https://github.com/pivotal/skenario