Thanks for the example of a Playwright report page.
I agree that getting browser tests (not even just visual tests) to work reliably is considerable work. I built out a suite at work for a rather complex web application and it certainly looks easier than it is.
A couple of notes:
- I disagree that you need a powerful VPS to run these tests, we run our suite once a day at midnight instead of on every commit. You still get most of the benefit for much cheaper this way.
- We used BrowserStack initially but stopped due to flakiness. The key to getting a stable suite was to run tests against a local nginx image serving the web app and wiremock serving the API. This way you have short, predictable latency and can really isolate what you're trying to test.
- I disagree that you need a powerful VPS to run these tests, we run our suite once a day at midnight instead of on every commit. You still get most of the benefit for much cheaper this way.
- We used BrowserStack initially but stopped due to flakiness. The key to getting a stable suite was to run tests against a local nginx image serving the web app and wiremock serving the API. This way you have short, predictable latency and can really isolate what you're trying to test.