aaronsteers

Great point. Many of Airbyte's customers are doing just that - adding new sources to their warehouses - like Google Drive, Gong, and a ton of sources that weren't as interesting previously for data analytics. But this creates a ton of work for the data engineering teams - to not only load all that extra data, but to deal with rate limits and then to conform the schemas into a usable format after loading.

For now, I think its 100% appropriate to think of the Context Store complementing the Warehouse and not replacing it per se. We're evaluating future integration options between the new Context Store and the traditional data warehouse, but nothing we have publicly announced as of now. I think both approaches have their strengths and killer use cases.

It's a good question and I won't pretend to predict the future on this one. I will say, I think Airbyte Agents is in a good position because our core Data Replication product has always had to mitigate the impacts of rate limiting and cumbersome upstream APIs. The new Agents toolset gives you the ability to query the upstream APIs directly (read: as a passthrough) while also letting you bypass them entirely when your agent can answer its question via the Context Store directly. Time and feedback from our users will confirm, but I do think this gives our customers a good balance of control - when to query upstream directly and when to utilize the Context Store to work around API limitations - whether inherent or artificially enforced by the vendor.

Agreed 100% - we're still super early in this journey, gathering data from our own usage and from our customers' feedback.

Helpful feedback, thank you! And your instincts are spot on. As of now, we have API based search, with filter predicates and field selection in JSON. While we haven't published anything on the backend implementation, I can say it does use a cloud-native storage medium where the filters are indeed pushed down as SQL. We want to be careful about if/when we offer direct SQL access, specifically because SQL dialects can differ drastically and we wouldn't want to break consumers if/when we change which dialect(s) are supported.

That said, please stay tuned - and thank you again for this valuable feedback.

Hey, fair enough. (100% human here, btw.) I think I misread your original question to be asking "why do we need a service (whether accessed via API/SDK/MCP/etc.)" vs just having skills (markdown + scripts)".

If you are already leveraging skills as scripts and APIs in your skills, then you understand the distinction. I'll attempt to re-answer your question with now hopefully a better understanding:

I think Airbyte Agents helps your agent by giving access to data across any and all of the systems it may need to get data from, or write data to. While you could hit the service APIs directly (via REST/CLI/etc.), in practice we find that not all use cases are amenable to this. Airbyte Agents does have REST APIs as well as SDKs and of course the MCP interface - so it's not really about MCP tools specifically, more about how you can access the data. The Airbyte Agents interface also reduces the number of creds that the agent needs to handle, giving a single portal (with logging and audit capabilities) for all the actions your agent is taking.

Sorry for the red herring of skills-v-tools. Let me know if you have any additional questions!

Great question, @Tsarp - Skill and tools work great together. What we've found is that agents generally need both to achieve great results. We're actually not trying to replace skills, but to give them new super powers.

Are there any examples you've run into where skills were missing tools (or data) that they needed for a specific task?

Hi, @jessewmc. Thanks for your reply. Regarding your points:

> If I'm reading correctly, the indexing (Context Store) is neutral/unopinionated? How does it select fields for indexing?

While we haven't yet published details on the backend implementation, I can say that our implementation performs very well without needing to prioritize specific fields for indexing. We aim for large text fields to perform decently and retrieval based on small/compressible fields like ints to be fast. (More to come on this in the coming months.)

> Have you done any testing on guided indexing, or metadata layers on top of the data?

We've been testing with different data scales and shapes. Nothing detailed to share yet, but performance has (so far) never itself become the bottleneck in our agent testing. (The LLM thinking itself is often the bottleneck.)

> My experience so far on similar work is that getting data in front of an agent isn't enough context to get useful/reliable answers enough of the time.

Airbyte has rich metadata on our upstream connector's data models, which I think helps us a lot to deliver helpful context to the agent. Another option, when optimizing for specific use cases, is to build your own agent tools on top of our Agent SDK. This allows you to make the calls organic and build the tools in a way that makes natural sense to the agent, regardless of source shape or which system(s) that data is coming from.

> This does look like a good foundation for that kind of tooling though!

We agree! Thanks again for sharing your thoughts here.

Hey, swyx! Great seeing you here.

> airbyte agents could serve as a form of MCP gateway

Exactly! And a single set of tools for agents to access both realtime (direct reads/writes) as well as cached (Context Store), bringing hopefully the best access path for each different use case.

> would love a "data engineering for ai engineers" type braindump ... at AIE

Great idea - we have a booth at AIE, and we'll submit there for a talk. Mario will reach out to you about this. :)

Glad to hear this resonates with you also. We're aiming to give agents more control over their context, and easier access paths regardless of the source system.

+1

Working with APIs is often frustrating and the worst ones are terribly ineficient and frustrating. Our Agent SDK and Agent Context Store insulates you and your agent from this headache, allowing you to query from those synced datasets directly.

The feedback about wanting to download a parquet file is super interesting...

Thanks! Really appreciate the kind words. Looking forward to seeing what our amazing community builds with these new tools.

That's great to hear - great minds think alike!

> give the agent access to the DB

This is where Airbyte really can shine, I think, and the total can be more the sum of the parts. Because Airbyte excels at data replication already, we can populate your the Agent Context Store without users or agents ever needing to think about the words "ELT" or "ETL".

We're listening carefully to feedback so we hope you will give it a try and let us know how it goes! Thanks!

Hello, Jared! Small world! Yes, we did deprecate our old PbA (Powered by Airbyte) offering, but in many ways our new Agents and Embedded offering is a more robust and agent-friendly successor to that older offering.

I am happy to hear you are still getting value out of PyAirbyte! If you do try out Airbyte Agents, please let us know how it goes! We are always listening to feedback and would love to hear from you as you explore the new tools and capabilities.

AJ here, from Airbyte.

Yes, we've definitely found that some API data models are easier for models to navigate than others.

The largest factors of Agent inefficiency we've identified so far are: 1. Many APIs lack robust-enough search, forcing agents to page through hundreds or thousands of paginated responses until they find the record they are looking for (our Context Store addresses this). 2. Many APIs have HUGE response sets. Our MCP helps handle this by letting the agent decide exactly what fields they can return. 3. With our SDK, you can literally build your own MCP on top of any source we support (50+ right now and will grow). This is super powerful, and allows you to build more ergonomic MCP servers and tools - even if the models themselves are not intuitive or easy for the LLM to leverage directly.

Combining all three of these together, we see the vast majority of challenges can be addressed via a strong system prompt for guidance. Fine tuning could get you further but anyway, you'd still want your fine tuned model to build on this same foundation, since the efficiences will transfer across use cases and models.

@ecares - Does this answer your question? What do you think?