> Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now.
I built a UAT agent on top of claude-agent-sdk, it uses Playwright and can spin up a preview instance for PRs we open. It uses its knowledge of the code to create a test plan, runs that test plan, and takes screenshots as it does so. On a recent PR I made a change to our MFA implementation and assumed that I would need to test it myself since MFA requires setup (and a TOTP device!) but no, the agent drove Playwright to change the instance configuration to enable MFA, created a user with MFA enabled, wrote the tiniest (handful of lines) Python TOTP and used that as the device and successfully verified that MFA still worked as expected. I did not expect that ability, and I wrote it! It was also exhaustive in its probe of functionality, way more than I had planned to do, even though I had planned to do a lot given the critical nature of MFA.
I built a UAT agent on top of claude-agent-sdk, it uses Playwright and can spin up a preview instance for PRs we open. It uses its knowledge of the code to create a test plan, runs that test plan, and takes screenshots as it does so. On a recent PR I made a change to our MFA implementation and assumed that I would need to test it myself since MFA requires setup (and a TOTP device!) but no, the agent drove Playwright to change the instance configuration to enable MFA, created a user with MFA enabled, wrote the tiniest (handful of lines) Python TOTP and used that as the device and successfully verified that MFA still worked as expected. I did not expect that ability, and I wrote it! It was also exhaustive in its probe of functionality, way more than I had planned to do, even though I had planned to do a lot given the critical nature of MFA.
Total token price for the UAT run: $5.69.