Could you do the stupid thing and have openai et al offer an API that, given text (or a hash of some text) returns whether that hash was generated by the service? Some 3rd party companies specialize in cheat detection and pay for the API then get paid by schools or whoever to detect detect cheaters.
Difficulties I can think of:
* Getting ai companies to offer this. I don't think it comes with a downside for them really though. You couldn't use it to actually retrieve results in any way, and you wouldn't have prompts.
* This only detects exact matches, people changing the output would defeat it. Fixable with some kind of fuzzy search that returns a 'distance to nearest response' but this obviously makes it more expensive and difficult to run, use, etc.
* People could still run the model themselves, as models get better and more expensive maybe this becomes less of a problem. Or maybe models get small enough while still generating good output that it becomes more of a problem. Who knows.
At least this would avoid AI grading AI issues
Difficulties I can think of:
* Getting ai companies to offer this. I don't think it comes with a downside for them really though. You couldn't use it to actually retrieve results in any way, and you wouldn't have prompts.
* This only detects exact matches, people changing the output would defeat it. Fixable with some kind of fuzzy search that returns a 'distance to nearest response' but this obviously makes it more expensive and difficult to run, use, etc.
* People could still run the model themselves, as models get better and more expensive maybe this becomes less of a problem. Or maybe models get small enough while still generating good output that it becomes more of a problem. Who knows.
At least this would avoid AI grading AI issues