jeffreyip

If you're using DSPy, you can also include it directly in this custom metric from the link above. It's hard for me to say 100% if there are advantages of doing this within DeepEval, but 8/10 times running evals in our ecosystem brings you more benefits than drawbacks. Let me know if you have trouble setting up!

Comment · 1 year ago

Definitely, feel free to join our discord for any questions on it: https://discord.com/invite/a3K9c8GRGt

Comment · 1 year ago

Do check it out, the early feedback has been great: https://docs.confident-ai.com/docs/metrics-dag

Comment · 1 year ago

Hey yes would definitely love to, my contact info is in my bio, please drop me an email :)

Comment · 1 year ago

Interesting, how are you remixing the order of questions? If we're talking about an academic benchmark like MMLU, the questions are independent of one another. Unless you're generating multiple answers in one go?

Do do synthetic data generation for custom application use cases. Such as RAG, summarization, text-sql, etc. We call this module the "synthesizer", and you can customize your data generation pipeline however you want (I think, let me know otherwise!).

Docs for synthesizer's here: https://docs.confident-ai.com/docs/synthesizer-introduction, there's a nice "how does it work" section at the bottom explaining it more.

Comment · 1 year ago

It's actually langfuse.com! Our quickstart walks you through the whole process: https://docs.confident-ai.com/confident-ai/confident-ai-intr...