Totally agree, especially about the need for well-architected, high-impact tests that go beyond just coverage. At Loadmill, we found out pretty early that building AI to generate tests was just the starting point. The real challenge came with making the system flexible enough to handle complex customer architectures. Think of multiple test environments, unique authentication setups, and dynamic data preparation.
There’s a huge difference between using an LLM to crank out test code and having a product that can actually support complex, evolving setups long-term. A lot of tools work great in demos but don’t hold up for these real-world testing needs.
And yeah, this is even trickier for higher-level tests. Without careful design, it’s way too easy to end up with “dumb” tests that add little real value.
There’s a huge difference between using an LLM to crank out test code and having a product that can actually support complex, evolving setups long-term. A lot of tools work great in demos but don’t hold up for these real-world testing needs.
And yeah, this is even trickier for higher-level tests. Without careful design, it’s way too easy to end up with “dumb” tests that add little real value.