Ask HN: How are you testing your LLM applications?

I'm looking to test an LLM application that uses a combo of LLM calls, Tool calls, code, and RAG. Testing will be both in production (errors, regressions) and in development (isolate a part of the system, iterate, run experiments). Would you use a platform or roll your own solution?

LangSmith looks interesting with tracing+playground, but I'm not using LangChain. I'm seeing established companies like Weights & Biases and Arize, but they seem more model focused than application focused. I'm also seeing startups like Parea and Scorecard that could be interesting.