richardblythman
Joined 6 years ago
Karma
34
Submissions
19
Member since
Nov 10
Recent submissions
Comment · 10 months ago
Comment · 1 year ago
Mainly at the maintainers. In the generated reports, we highlight issues and suggest improvements to the docs (working on improving the reports as we run across more libraries).
Comment · 1 year ago
Last week we made it to the front page with our post about benchmarking how well coding agents interact with libraries and APIs. The response was positive overall, but many wanted to see the code.
For those just catching up: The problem is that existing benchmarks focus on self-contained codegen. StackBench tests how well AI coding agents (like Claude Code, and now Cursor) use your library by: • Parsing your documentation automatically • Extracting real usage examples • Having agents generate those examples from a spec from scratch • Logging every mistake and analyzing patterns
You can find out more information about how it works and how to run it in the docs https://docs.stackbench.ai/
Next up, we’re planning to add more: • Coding agents • Ways of providing docs as context (e.g. Mintlify vs Cursor doc search) • Benchmark tasks (e.g. use of APIs via API docs) • Metrics
We're also working on automating in-editor testing and maybe even using an MCP server.
Contributions and suggestions very welcome. What should we prioritize next? The issues tab is open.
Comment · 1 year ago
I think part of the reason i like Claude Code so much for non-technical organizational work, is that it's like a controllable memory with human in the loop (as well as a powerful general agent in the loop). Most of what i do is basically just doc creation and management. "Create this doc using this context and store it in this directory." In ChatGPT, I don't have much control or visibility over the memory.
Comment · 1 year ago
This resonates. Upskilling to AI tools is perhaps the biggest problem of our day. One idea we have to tackle this problem is to bring onboarding/learning directly into the user's work environment, track struggles and offer targeted support, and create continuous feedback loops. If anyone has faced challenges with increasing activation and retention of users on pilots (or external-facing products), would love to chat and see how we can help .
Comment · 1 year ago
i think you prob need a bit of both. would you add llms.txt to your docs or make it crawl like a human?
Comment · 1 year ago
do we want every coding agent writing everything from scratch for every project?
or would reusability/modularity across projects and teams be beneficial?
Comment · 1 year ago
i mean that we wanted an email address to send the results to when they finish.
based on comments here, i do think we should allow users to run the audit first (and provide an email address if they want us to follow up with results later).
Comment · 1 year ago
im paraphrasing. the questions i asked to dev tool builders were more neutral.
Comment · 1 year ago
good to know. i think we're likely to move towards running coding agents locally in the next iteration.
Comment · 1 year ago
we send out an email when the tests are finished (takes about 30 mins)
Comment · 1 year ago
If coding agents are the new entry point to your library, how sure are you that they’re using it well?
I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know.
Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation.
So we built a simple app to test how well coding agents interact with libraries: • Takes your library’s docs • Automatically extracts usage examples • Tasks AI agents (like Claude Code) with generating those examples from scratch • Logs mistakes and analyzes performance
We’re testing libraries now, but it’s early days. If you're interested: Input your library, see what breaks, spot patterns, and share the results below.
We plan to expand to more coding agents, more library-specific tasks, and new metrics. Let us know what we should prioritize next.
Building a meal database: I add meals to a directory using whatever's convenient: photos from a cookbook, or copy-paste from PDFs or websites. I modify the recipes as needed (e.g. scale 2x for lunch leftovers, swap out ingredients that are hard to get, add more information for soft boiling vs hard boiling an egg). The agent scrapes the nutritional information for the recipe automatically via the USDA API and stores it.
Weekly planning & Nutritional analysis: When it's time to plan my week, I can either select from my meal library or let the agent suggest combinations. It then generates a shopping list broken down by category (dairy, vegetables, etc). The agent creates detailed nutrition reports showing my daily protein, micronutrients, etc. Currently, I am still manually filling my shopping cart online from the generated list (but working on automating that too).
Micronutrient tracking: I was recently researching whether I needed to invest €100/month in AG1, but after analyzing my current meal plan, it turns out I'm already getting most of my vitamins. The agent suggested a few strategic tweaks (adding sunflower and pumpkin seeds to my bowls for Vitamin E and Zinc, and a kiwi after dinner for vitamin C) to get me where I need to be. I realized I probably do need to take some Vitamin K2 supplements as it's hard to get from food here.
Symptom investigation: I've always struggled with digestive issues but could never narrow down the cause. I've been tracking symptoms using the Bristol scale (so-called log agent!) along with eczema flare ups, and from 2 weeks of data our first hypothesis is that tomatoes are the issue. Apparently tomatoes are a "nightshade" (hadn't heard of it). Starting Monday, we're running a structured elimination diet and the agent has me on plain salmon/chicken, rice and spinach until we kick off.
Here's a demo of how I interact with it (via an editor we made for non-coding claude code agents). https://www.youtube.com/watch?v=ShcYuV7Evvw
Still early, but AI allows me to use the time I used to spend on planning meals and creating shopping lists for higher level analysis. I'm still very much in the loop. It's no good at data analysis without me, but with it my capabilities reach much further.