adinagoerres
Joined 3 years ago
Karma
52
Submissions
22
Member since
Mar 19
Recent submissions
Comment · 3 months ago
omg I got it right!!!
Comment · 6 months ago
awesome to hear!
Comment · 6 months ago
couldn't agree more
Comment · 6 months ago
setup guide is here: https://docs.superglue.cloud/mcp/using-the-mcp
best used for granular access management to integrations and data, in agent applications
Comment · 6 months ago
Yes, we've seen this work best with clear instructions around Salesforce, Attio etc. superglue does come with a number of observability and monitoring features, but of course this can be extended for a specific setup.
Comment · 6 months ago
Adina here, one of superglue's creators. I'm curious to hear folks opinions on this. Also to clarify how agents are able to call APIs via superglue: the first step is to set up auth and systems on superglue so it can process and extract documentation and any other context for calling the APIs, which is then passed on to the agents
Comment · 11 months ago
How well can agents work with APIs they’ve never seen before? We tested 41 APIs across 8 different LLMs to find out.
API execution is great for benchmarking, because it tests core qualities and limitations of LLMs:_the depth of the data they were trained on, their stateless architecture, context dependency, and reasoning.
Today we're releasing v2 of API-Bench:_a benchmark that tests how well LLMs can execute against APIs. Here are the results: https://superglue.ai/benchmark_v2
Tl;dr:_LLMs fail at integrations because they lack ground truth, lack state, lack debugging ability, and lack access to real system context - everything API integrations fundamentally require.
Here’s what we found:
1. LLMs are only as good as the data they’re trained on:_when docs change, APIs evolve, or systems are niche/long-tail, they use outdated patterns, guess missing pieces and hallucinate endpoints and parameters.
2. LLMs are stateless, but integrations are stateful:_auth handshakes, pagination, retries, multi-step flows all need memory but LLMs can’t persist intermediate values or reason across steps.
3. LLMs produce code that “looks right” but fails at runtime: LLMs cannot isolate the failing step and understand real error messages, so they can’t change what’s broken or retry with new hypotheses.
4. LLMs can’t reliably interpret imperfect API design:_humans can infer the intended function, LLMs will hallucinate what looks reasonable.
We open sourced the benchmark so you can test your own APIs or contribute new ones: https://github.com/superglue-ai/superglue/tree/main/eval/llm...
Curious to hear your experience, and of course always happy to share more learnings.
Comment · 11 months ago
creator of superglue here. we are trying to crack one of the hardest problems in infra management, with the goal to get rid of nasty systems and expensive consulting projects. always looking for the gnarliest of systems and how to make them future-compatible.
Comment · 1 year ago
self-reflection is very important for both humans and LLMs, indeed
Comment · 1 year ago
Hey HN, I'm Adina, Stefan's co-founder at superglue. When we started working on LLM-powered integrations about a year ago, the models were barely good enough to handle simple mappings. We started benchmarking our performance as an internal evals project and thought it would be fun to open source it, to create more transparency around LLM performance. Our goal here is to understand how we can make agents production-ready and improve reliability across the board.
Comment · 1 year ago
Hi HN! Adina here from superglue. Today I’d like to share a new benchmark we’ve just open sourced: an Agent-API Benchmark, in which we test how well LLMs handle APIs.
tl;dr: LLMs suck at writing code to use APIs.
We ran 630 integration tests across 21 common APIs (Stripe, Slack, GitHub, etc.) using 6 different LLMs. Here are our key findings: - Best general LLM: 68% success rate. That's 1 in 3 API calls failing. Would you ship that? - Our integration layer scored a 91% success rate, showing us that just throwing bigger/better LLMs at the problem won't solve it. - Only 6 out of 21 APIs worked 100% of the time, every other API had failures. - Anthropic’s models are significantly better at building API integrations than other providers.
What makes LLMs fail hard: - Lack of context (LLMs are just not great at understanding what API endpoints exist and what they do, even if you give them documentation which we did) - Multi-step workflows (chaining API calls) - Complex API design: APIs like Square, PostHog, Asana (Forcing project selection among other things trips llms over)
We've open-sourced the benchmark so you can test any API and see where it ranks: https://github.com/superglue-ai/superglue/tree/main/packages...
Check out the repo, consider giving it a star, or see the full ranking at https://superglue.ai/api-ranking/
If you're building agents that need reliable API access, we'd love to hear your approach - or you can try our integration layer at superglue.ai.
Next up: benchmarking MCP.