I'm working on a rather naive approach that is focused on identifying errors in a LLM response by using LLMs. What I can share right now are screenshots with regards to how it works. The basic idea is you can use other high-quality models to validate and compare against to find irregularities or errors. You can see what it looks like below:
The basic idea behind my chat system is, every model can be wrong, but it is unlikely that all will be wrong at the same time. This chat system is based on what I've learned when building my spelling and grammar checker. If you look at the following links, you can see that even the best models can get it wrong, but it is unlikely that others will get it wrong at the same time.
https://app.gitsense.com/--/images/options.png
https://app.gitsense.com/--/images/validate.png
https://app.gitsense.com/--/images/models.png
The basic idea behind my chat system is, every model can be wrong, but it is unlikely that all will be wrong at the same time. This chat system is based on what I've learned when building my spelling and grammar checker. If you look at the following links, you can see that even the best models can get it wrong, but it is unlikely that others will get it wrong at the same time.
https://app.gitsense.com/?doc=6c9bada92&model=GPT-4o&samples...
https://app.gitsense.com/?doc=905f4a9af74c25f&model=Claude+3...