I had similar results trying out Amazon Q. I fed it a copy of our employee handbook and had it scrape our corporate website. My idea is that this could be a good fit for an HR bot.
I asked it a few simple fact based questions, such as who our CEO is. It answered with our previous CEO based on an old press release. I asked it what year it is, and it couldn’t say. A few other specific fact based questions were either wrong or unanswered.
So I asked it higher level questions about what the corporation does. It was able to answer those.
Finally, I began asking questions about policies in the handbook. When I asked about harassment, it refused to provide any answers. It felt like I was filtered. Filtering will kill the utility of these products. Something like a legal practice is messy business that the filter won’t like.
I asked it a few simple fact based questions, such as who our CEO is. It answered with our previous CEO based on an old press release. I asked it what year it is, and it couldn’t say. A few other specific fact based questions were either wrong or unanswered.
So I asked it higher level questions about what the corporation does. It was able to answer those.
Finally, I began asking questions about policies in the handbook. When I asked about harassment, it refused to provide any answers. It felt like I was filtered. Filtering will kill the utility of these products. Something like a legal practice is messy business that the filter won’t like.