yohji1984
Joined 5 months ago
Karma
8
Submissions
15
Member since
May 29
Recent submissions
Comment · 3 months ago
Sorry I will edit the post
Comment · 3 months ago
Hi HN, I've been working on an agentic runtime framework. The earlier benchmark eval is promising. But I can see the limits of the test design and the fragility of the runtime itself. I would like to ask for your reviews of the framework and the eval process itself.
https://github.com/Tura-AI/tura
https://turaai.net/docs#benchmark-current-test-set-record
Tura-AI/tura https://turaai.net/blog#why-i-am-building-tura
Comment · 4 months ago
Sorry, I missed the Open Items section. You're right about that, designing a good eval harness can be difficult and expensive. Maybe we need some kind of community project for agentic evals, where people can share eval harnesses and run logs.
Comment · 4 months ago
I'm wondering why all these token-saving solutions focus their benchmarks exclusively on simple Q&A tasks. If their tools truly saved money in real, long-term programming tasks, they would have definitely published those benchmark results instead of just Q&A tests, especially since a simple code editing benchmark with a hidden eval harness is very easy to design. Personally, asking a coding agent questions without any code editing is a very rare case for me