Kappa90
Joined 9 years ago
Joined 9 years ago
The annotation/semantic layer agent creates a new description of the schema on sync, which represents the current state, but that includes stale columns as of today, data is not dropped.
I’ll implement automated schema migrations in the next week or so!
80% of the benchmark questions are aggregations, 16% are multi-hop, 4% are lookups/subqueries.
Multi-hop is where LLMs struggle the most (hallucinations, partial answers), and aggregations is where you get the most token efficiency, since you skip on pagination which you need with APIs/MCPs that don't provide filters.
Of the small models I tested, Qwen 3.5 is the clear winner. Going to larger LLMs, Sonnet and Opus lead the charts.
There's 75 questions, divided in 5 use case groups: revenue ops, e-commerce, knowledge bases, devops, support.
I then generated a synthetic dataset with data mimicking APIs ranging from Stripe to Hubspot to Shopify to Zendesk etc..
I expose all the data through Dinobase vs. having one MCP per source e.g. one MCP for Stripe data, one MCP for Hubspot data etc.
I tested this with 11 models, ranging from Kimi 2.5 to Claude Opus 4.6.
Finally there's an LLM-as-a-judge that decides if the answer is correct, and I log latency and tokens.
I just published the first episodes today and have zero listeners or subscribers, so it can only grow from here!
https://historyofthemajorityworld.com
Spotify: https://open.spotify.com/show/4qJh2jNwMvWcLP1J1EMXxr?si=dcc7...
Apple Podcasts: https://podcasts.apple.com/rw/podcast/history-of-the-majorit...
The main reason why we advocate for a single-loop is because subagents suffer from the telephone game problem: context is lost when the interface is just an agent-to-agent prompt.
We are still working on subagents for parallelization, but if you can stick to the main loop, everything works much better.
Found out two weeks later when angry users couldn't log in during peak hours. Damage: 4 hours of downtime, one very grumpy security team, and a new "no AI for auth code" policy.
Disclaimer: I’m the developer behind CodeBeaver
It aggregates the news from thousands of newspapers, youtube channels and podcasts, then serves you a summarized digestible version, as a feed and as a personalized podcast, using LLM models.
You can also ask to clarify or explain the news, and there's a new function to ask if an article's claims are supported by other sources, to avoid reading biased or opinionated information.
Disclaimer: I'm one of the founders of Tailor
Right now I’m working on including automatic fact checking and insights on how each source might be opinionated vs. reporting just the facts.
The idea is that Saga helps you resurface connections and ideas without having to do the manual work yourself.
This allows for a more serendipitous discovery of knowledge while you're crafting content or researching a topic.
Areas where we see great promise are: - analyzing interviews and notes for product and user research - automatically generate company or support wikis from docs - discover insights while writing meeting/team notes and documentation
Currently beta testing realtime collaborative editing and we'll start pushing towards integrating with other tools after that.
[0] https://saga.so