Use this GitHub action to easily integrate RAG system evaluation into your CI/CD process. The README contains full instructions and an example.
This action runs every time you create a PR and for each subsequent commit to the branch. Simply point it to a file that contains test questions and optional reference answers and context. It evaluates the questions and then scores the RAG generated response against the provided references. After it computes the scores across 6 different metrics, it generates a table that is displayed as a comment on your PR.
This action runs every time you create a PR and for each subsequent commit to the branch. Simply point it to a file that contains test questions and optional reference answers and context. It evaluates the questions and then scores the RAG generated response against the provided references. After it computes the scores across 6 different metrics, it generates a table that is displayed as a comment on your PR.