davinchia

Haha indeed!

On a more serious note, just as swyx mentioned in a comment further up, we do believe a lot of the challenges of reliably operationalising agents boil down to data. All of which is non-obvious to AI engineers (besides Frontier Labs gathering/generating data for model training).

What the right shape is - we are all figuring it out. Happy to trade notes.

Sorry for that experience. We had a bad billing support routing issue and it’s since been fixed. Thank you for calling it out. We'll aim to do better!

Yeah, lots of blood sweat and tears here!

Let's gooooo!

Since this is executed from Java code, we did not want to have to bundle kubectl (I'm assuming that's what you are referring to) with the Jar. We also did not want to run an exec command.

Are you asking why not GRPC between the various job pods?

Airbyte engineer here. We use Dbt to normalise the delivered data at the final destination warehouse.

If it's not a destination warehouse, how transformations are handled depends on the connector implementation.

Great points!

"If you know you're going to spend 15 hours on it, then you might as well use a system you know will cost less to extend or scale over time."

I wish younger me realised that earlier :)

"And maybe even a canned CI/CD pipeline in a container so they can start iterating on it immediately."

Definitely! Although a good number of users are surprisingly happy with their Airbyte instances on a single node.

We do have a Kubernetes offering for those looking to scale Airbyte beyond a single node. We also have Kustomise/Helm deploys for this, though I'll be the first to admit that the Helm charts are mostly community-maintained and can be improved. This is one of our (my) top priorities going into the next Quarter.

Airflow is a general orchestration tool that fit into the Python stack extremely well. It wasn't build to scale though, so once you want to run something more than once a second, you are going to be jumping through hoops.

My experience with Dataflow is 1.5 years old, so things might have changed, but I felt it more to be a unified, simplified Hadoop/Spark framework. It unifies the batch/streaming concepts but is still pretty low-level.

Within ELT, or ETL, Airflow/Dataflow can fulfills all 3 components.

Airbyte focuses just on EL (though we have basic T functionality around normalisation). Our intention is to leave T to the warehouse, since warehouses like Redshift/Snowflake/BigQuery are extremely powerful these days, and tools like DBT, give the users more more flexibility to recombine and consume the raw data than a specialised ELT pipeline.

In summary, I would say Airbyte is a specialised subset of Airflow/Dataflow, and it's possible to use Airbyte with either tools, though I'd guide someone towards DBT.

We are working on it. This should be out next quarter!