bbischof

This is great; it's honestly so f'in annoying working with documents with multiple users AND LLMs

Data visualizations and live dashboards of people tweeting that '_ is dead'

Label studio is fine if it covers your need, but in many cases the core opportunity in an eval interface is fitting in with the SME’s workflow or current tech stack.

If label studio looks like what they can use, it’s fine. If not, a day of vibecoding is worth the effort to make your partners with special knowledge comfortable.

10x speed up; what more do you want

We allude to 2 when talking about using explanations first, but I totally agree. One minor comment is explanations after can sometimes be useful for understanding how the model came to a particular generation during post-hoc evals.

Point 1 is also a good callout. I added something on this for llm judge but it’s relevant more broadly.

Bryan here, one of the authors.

Sure. Ultimately, you want to use KG to increase your ability to do great retrieval.

Why do graphs help with retrieval? Well, don’t overlook the classic pageant example: graphs provide signal about the interconnectivity of the docs.

Also, sometimes the graph itself are a kind of object you want to retrieve over.

Super interesting. I haven’t seen much chain support tech for ts folk which is a hard requirement for more production applications.