cschiller

By default, the SDKs use our API endpoints, where we run a combination of models to maximize accuracy and reliability. This also enables us to provide logging with screenshots and reasoning to help with debugging.

That said, we're currently experimenting with a few customers who run our tooling against their own hosted models. While it's not publicly available yet, we might introduce that option going forward.

Would love to hear more about your use case, if a self-hosted setup is relevant or just the use of your own LLM tokens?

Yes, you can use our SDKs to run it locally on Simulators, Emulators, and real devices. We also support popular third-party device farms via the WebDriver protocol.

Good call! The timing was actually a coincidence, but not unexpected. OpenAI had already announced their plans to work on a desktop agent, so it was only a matter of time.

From our tests, even the latest model snapshots aren't yet reliable enough in positional accuracy. That's why we still rely on augmenting them with specialized object detection models. As foundational models continue to improve, we believe our QA suite - covering test case management, reporting, agent orchestration, and infrastructure - will become more relevant for the end user. Exciting times ahead!

Thank you! Sonnet 3.5 is indeed a powerful model, and we're actually using it. However, even with the latest version, there are still some limitations affecting our specific use case. For instance, the model struggles to accurately recognize semi-overlaid areas, such as popups that block interactions, and it has trouble consistently detecting when UI elements are in a disabled state.

To address these issues, we enhance the models with our own custom logic and specialized models, which helps us achieve more reliable results.

Looking forward, we expect our QA Studio to become even more powerful as we integrate tools like test management, reporting, and infrastructure, especially as models improve. We're excited about the possibilities ahead!

Thanks for sharing your experience! Completely agree - there's often a huge gap between the perception that testing is "solved" and the reality of manual QA still being necessary, even for core features. We recently had a call with one of the largest US mobile teams and were surprised to learn they're still doing extensive manual testing because some use cases remain uncovered by traditional tools. It's definitely not as "solved" as many might think.

Thanks for your thoughtful response! Agree that digging into the root cause of a failure, especially in complex microservice setups, can be incredibly time-consuming.

Regarding writing robust e2e tests, I think it really depends on the team's experience and the organization’s setup. We’ve found that in some organizations—particularly those with large, fast-moving engineering teams—test creation and maintenance can still be a bottleneck due to the flakiness of their e2e tests.

For example, we’ve seen an e-commerce team with 150+ mobile engineers struggle to keep their functional tests up-to-date while the company was running copy and marketing experiments. Another team in the food delivery space faced issues where unrelated changes in webviews caused their e2e tests to fail, making it impossible to run tests in a production-like system.

Our goal is to help free up that time so that teams can focus on solving bigger challenges, like the debugging problems you’ve mentioned.

One of our customers recently compared GPTD with Maestro’s Robin (formerly App Quality CoPilot). Their mobile platform engineering manager highlighted three key reasons for choosing us: lack of frustration, ease of implementation, and reliability.

To be more concrete their words were: - “What you define, you can tweak, touch the detail, and customize, saving you time.” - “You don’t entirely rely on AI. You stay involved, avoiding misinterpretations by AI.” - “Flexibility to refine, by using templates and triggering partial tests, features that come from real-world experience. This speeds up the process significantly.”

Our understanding is that because we launched the first version of GPT Driver in April 2023, we’ve built it in an “AI-native” way, while other tools are simply adding AI-based features on top. We worked closely with leading mobile teams, including Duolingo, to ensure we stay as aligned as possible with real-world challenges.

While our focus is on mobile, GPT Driver also works effectively on web platforms.

I agree that it can seem counterintuitive at first to apply LLM solutions to testing. However, in end-to-end testing, we’ve found that introducing a level of flexibility can actually be beneficial.

Take, for example, scenarios involving social logins or payments where external webviews are opened. These often trigger cookie consent forms or other unexpected elements, which the app developer has limited control over. The complexity increases when these elements have unstable identifiers or frequently changing attributes. In such cases, even though the core functionality (e.g., logging in) works as expected, traditional test automation often fails, requiring constant maintenance.

The key, as to other comments, is ensuring the solution is good at distinguishing between meaningful test issues and non issues.

I would say that mobile apps are still the primary format for launching new consumer services, incl. new apps like ChatGPT and many others. However we’ve observed that teams are expected to do more with less—delivering high-quality products while ensuring compliance, often with the same or even smaller team sizes. This is why we focus on minimizing the engineering burden, particularly when it comes to repetitive tasks like regression testing, which can be especially painful to maintain in the mobile ecosystem due to use of third-party integrations (authentication, payments, etc.).

Yes, great point! We have an 'Assistant' feature where you can perform the flow on the device, and we automatically generate the test case as you navigate the app. As you mentioned, it’s a great starting point to quickly automate the functional flow. Afterwards, you can add more detailed assertions as needed. Technically we do this by using both the UI hierarchy from the app as well as vision models to generate the test prompt.

Thank you for it being so well said!

The beauty of it is that everyone is likely to face or notice problems in their own life which offer the chance to be solved. And usually one is not the only one facing it :)

Hopefully this inspires someone to solve a problem which nags them and which they can get passionate about.

We are excited about the opportunity to work on this problem. And are grateful for the constructive feedback we are receiving from the HN community.