pablomendes

As large language model (LLM) agents increasingly undertake digital work, reliable frameworks are needed to evaluate their real-world competence, adaptability, and capacity for human collaboration. Existing benchmarks remain largely static, synthetic, or domainlimited, providing limited insight into how agents perform in dynamic, economically meaningful environments. We introduce UpBench, a dynamically evolving benchmark grounded in real jobs drawn from the global Upwork labor marketplace. Each task corresponds to a verified client transaction, anchoring evaluation in genuine work activity and financial outcomes. UpBench employs a rubric-based evaluation framework, in which expert freelancers decompose each job into detailed, verifiable acceptance criteria and assess AI submissions with per-criterion feedback. This structure enables fine-grained analysis of model strengths, weaknesses, and instruction-following fidelity beyond binary pass/fail metrics. Human expertise is integrated throughout the data pipeline (from job curation and rubric construction to evaluation) ensuring fidelity to real professional standards and supporting research on human-AI collaboration. By regularly refreshing tasks to reflect the evolving nature of online work, UpBench provides a scalable, human-centered foundation for evaluating agentic systems in authentic labor-market contexts, offering a path toward a collaborative framework, where AI amplifies human capability through partnership rather than replacement.

In what kinds of workloads or usage patterns do you see the biggest performance gains vs traditional FaaS + storage stacks?

What does the tech stack look like?

Is the course focused on LLMs used to generate text or does it also talk about other kinds of testing like search, images, etc?

Cool! What's next in the roadmap?

Congrats on the launch! I'm an inveterate search nerd so I can't help but ask how did you implement search?

What are you using for the search tech?

That statement is both kind of true and, well, revisionist. Originally there was a strong focus on logics, clean comprehensive modeling of the world through large complicated ontologies, and the adoption of super impractical representation languages, etc. It wasn't until rebellious sub-communities went rogue and pushed for pragmatic simplifications that things got any widespread impact at all. So here's to the crazy ones, I guess.

One thing that works well for me is going working-to-working. Get a simple, de-scoped, incomplete, probably crappy version done end-to-end. Now it's not about finishing, it's about improving. And if it was worth building in the first place, it will beg for improvement. And then it's easier to just keep turning the crank, working-to-working.

Hey @afandian, can I interview you about the tech stack you all used to build search at Crossref.org?

Are you planning to implement search and filter tags?

How did you implement snippet search?