I built an in-house version of this a couple of years ago for where I was working. My concern would be that by excluding observability, you might end up creating a really selective dataset, whose conclusions you're then asking companies to take seriously when allocating resources to different possible roadmaps.
My guess would be that agent logs would highlight obvious feature requests and bugs for smaller companies - like customers expecting an AI video editor product to be able to add subtitles to a video by itself.
For larger companies who deal with a higher volume of inbound customer support / agent requests, there will probably be big, noisy, already-known-by-the-team query clusters that make up big portions of the dataset - for example, "billing issue with my subscription". After those big clusters you'll likely have a really long tail of different queries, and - without deep observability - no real way to rank their importance. I also think you'd be unlikely to understand the root cause of the product issue in a complex developed product with lots of users solely from agent logs. Most product teams can't make good product decisions consistently, and they're working with a lot more data.
If coupled with staying out of evals (which, btw, I wouldn't find trust-building, if I were a potential customer of yours), I think that it might be difficult to provide genuine value in this space for larger orgs - without evals it's easily dismissed as just fancy & mostly-contextless sentiment analysis.
But I hope I'm wrong! I do think that (though each org's needs probably have to be catered to in a very boutique way) there are huge gains available by rolling LLMs & language analysis into existing product workflows, and that what you're pitching is absolutely a part of what companies should be doing. We are, of course, meant to actually listen to customers - and LLMs/agents should be making that easier, not harder. Absolute best of luck!
My guess would be that agent logs would highlight obvious feature requests and bugs for smaller companies - like customers expecting an AI video editor product to be able to add subtitles to a video by itself.
For larger companies who deal with a higher volume of inbound customer support / agent requests, there will probably be big, noisy, already-known-by-the-team query clusters that make up big portions of the dataset - for example, "billing issue with my subscription". After those big clusters you'll likely have a really long tail of different queries, and - without deep observability - no real way to rank their importance. I also think you'd be unlikely to understand the root cause of the product issue in a complex developed product with lots of users solely from agent logs. Most product teams can't make good product decisions consistently, and they're working with a lot more data.
If coupled with staying out of evals (which, btw, I wouldn't find trust-building, if I were a potential customer of yours), I think that it might be difficult to provide genuine value in this space for larger orgs - without evals it's easily dismissed as just fancy & mostly-contextless sentiment analysis.
But I hope I'm wrong! I do think that (though each org's needs probably have to be catered to in a very boutique way) there are huge gains available by rolling LLMs & language analysis into existing product workflows, and that what you're pitching is absolutely a part of what companies should be doing. We are, of course, meant to actually listen to customers - and LLMs/agents should be making that easier, not harder. Absolute best of luck!