We are working on an AI product in a highly regulated industry (Investing).
Recently we have been experimenting using the GPT3 API as a Junior Equity Analyst. On an eyeball check the results of the technology are impressive.
The problem is that there is no way to validate the feedback on scale. I.e., we can't receive statistics about the feedback from the API.
In contrast, for our own Entity Recognition models we can (and do) calculate probabilities that explain why a certain entity is shown.
Hence, I think for API users of GPT3, OpenAI should return additional statistics why a certain result is returned the way it is to make it really useful and more importantly compliant.
The problem is that there is no way to validate the feedback on scale. I.e., we can't receive statistics about the feedback from the API.
In contrast, for our own Entity Recognition models we can (and do) calculate probabilities that explain why a certain entity is shown.
Hence, I think for API users of GPT3, OpenAI should return additional statistics why a certain result is returned the way it is to make it really useful and more importantly compliant.