jtagliabuetooso
Joined 7 years ago
Karma
22
Submissions
37
Member since
Mar 26
Recent submissions
Comment · 6 months ago
How is the Git semantics of merge, rebase and diff defined in the system?
Comment · 6 months ago
“Why it’s distinctive” is misleading (perhaps LLM-generated)?
Imo, we cite other work because it puts our work in context for experts and beginners alike; because it makes clear that we all stand on someone else’s shoulders (progress is, most of the time, a collective endeavor, not a lone-genius affair); because it is intellectually honest to acknowledge our debts.
Especially today, when putting research ideas out there almost guarantees they will be plagiarized by someone vibe-coding or vibe-writing, recognizing that our contributions come from somewhere is more important than ever. The implementation may or may not be novel, but the fact that it depends on LLMs even in the README should make you even more aware of why proper attribution is crucial: what's the incentive for open innovation if we all behave like this?
Comment · 6 months ago
Cool release.
IMO, "Why it's distinctive" is a bit misleading on a few points: certainly the dbt and DX folks can add their POV, but even considering stuff I know / authored ;-), https://arxiv.org/pdf/2308.05368 from 2023 (and following releases) cover branches in a native way (no clone), immutability (re-run), and lineage.
Extensions to be considered are different languages (what about Python), and branch semantics. Two immediate questions would be: can you nest branches? How does merge works across systems if you don't control compute?
Comment · 6 months ago
Data contracts as types and compile time checks (even across languages) are not new - this is a recent paper exposing the idea of correctness-by-design pipeline, which is a super set of this particular issue obviously (disclaimer: I'm one of the author of the paper): https://arxiv.org/pdf/2602.02335
Comment · 1 year ago
Author here: happy to answer questions on our experience with duckDB / DF as part of a larger system running OLAP workloads.
Comment · 1 year ago
Marimo and astral are great and we use them both, they are not however infrastructure companies, so the parallel is a bit imperfect. Wouldn't you use AWS because it's closed source? And BigQuery? Or Motherduck?
There is no "one size fits all" when it comes to building companies, and the right answers depend on many factors: it would be interesting to know your choices for example!
I do agree with you that is important to give back to the ecosystem, but per size / dollar, bauplaners have done and continue doing as much as anyone. All in all, we have shared our ideas in the community in 50+ research papers in top venues (with thousands of citations), and we have quite a few popular open source contributions, with millions of downloads and >10k GitHub stars in total (our FaaS scheduler simulator was just open sourced with our VLDB25 WS paper).
You can be a good citizen of the Python / AI / database ecosystem without doing open source as a business strategy: the reality is more nuanced I believe!
Comment · 1 year ago
We are not proposing or advocating for any approach to development (I personally almost never use notebooks these days and run Bauplan with preview).
The blog together with our marimo friends is to showcase that you can have notebook development if you like it AND cloud scaling (which u need) without code changes, thanks to the fact that both marimo and Bauplan are basically Python (maybe a small thing, but there is nothing else in the market remotely close).
On the AI part, we agree: the fact that bauplan is just Python, including data management and infra-as-code, makes it trivial for AI to build pipelines in Bauplan, which is not something that can be said about other data platforms - if you follow our blog, we are releasing in a few weeks or so a full "agentic" implementation with Bauplan API of production ETL workloads, which you may find interesting.
Comment · 1 year ago
Hey Ben, thanks for your message.
We have people building stuff featured here (https://www.bauplanlabs.com/build-with-bauplan) as well as online (e.g. https://blog.det.life/bauplan-the-serverless-data-lakehouse-...), plus of of course our examples repo in Github that you can check as part of the tutorial.
Our largest client is a 5BN / USD year company running thousands of jobs on bauplan. If you have something in mind, you can try out the public sandbox for free and come on our Slack, and I'm happy to build something with you.
Comment · 1 year ago
Thanks for the comment: your frustration is the default in the industry, and it's part of the reasons why Bauplan was built.
"but it always takes forever and I'm never even able to articulate why." -> there are way more factors at play than DoFeatures unfortunately, see for example Table 1 (https://arxiv.org/pdf/2404.13682). Even knowing which data people have developed on is hard, which is why bauplan has git-for-data semantics built in: everyone works on production data, but safely and reliably, to avoid data skews.
Each computer is different, which is why bauplan adopt FaaS with isolated and fully containerized functions: you are always in the cloud, so no skew in the infra etc.
The problem of "going to production" is still the biggest issue in the industry, and solving it is not a one-fix kind of thing, but unfortunately the combination of good ergonomics, new abstractions and reliable infra.
Comment · 1 year ago
Importantly, kedro does not run things for you, resulting in a suboptimal experience because the runtime and dsl are separated: in particular, it does not solve the problem of having K different systems with scattered logs and not easy to integrate APIs.
If you want to dive deeper in one line reproducibility, you can chek our SIGMOD24 paper: https://arxiv.org/pdf/2404.13682. Let us know what you think!
Comment · 1 year ago
Thanks for your comment. As stated elsewhere, we understand the need for people to know how the system works, and have contributed back our ideas (and quite a bit of open source code) to the community: if you want to check our blogs and / or papers, I'm sure you'll find many interesting things.
If you're worried about data movement or secure deployment, none of that is an issue because of Iceberg + BYOC option.
Databricks and Snowflake, just to mention two players in a similar space, are not OS: did you feel that would prevent you from adopting them as well?
Comment · 1 year ago
Spark is technically not Python, even if we support PySpark with the relevant decorator but it's a very niche use case for us.
As for all the other Python packages, including proprietary ones, the FaaS model is such that you can declare any package you want in a function as node in the pipeline DAG, and any other in another: every function is fully isolated, and you can even selectively use pandas 1 in one, pandas 2 in another, or update the Python interpreter only in node X.
If you're interested in containerization and FaaS abstractions, this is good deep dive: https://arxiv.org/pdf/2410.17465
If you're more the practical type, just try out a few runs in the public sandbox which is free even if we are not GA.
Comment · 1 year ago
Thanks for checking out bauplan (which also supports BYOC, so I guess it is indeed hostable by you in a sense!).
We've done quite a lot of open source in our life, at Bauplan (you can check our github), and before (you can check me ;-)), so the comment seems unfair!
We understand the importance of being clear on how the platform works, and for that we have a long series of blog posts and, if you're so inclined, quite a few peer-reviewed papers in top conferences, ranging from low-level memory optimizations (https://arxiv.org/abs/2504.06151), columnar caching (https://arxiv.org/abs/2411.08203), novel FaaS runtimes (https://arxiv.org/pdf/2410.17465), pipeline reproducibility (https://arxiv.org/pdf/2404.13682) and more.
We are also always happy to chat about our tech choices if you're interested.
Comment · 1 year ago
Yeah, terms are confusing sometimes! "Data lakehouse" is weirdly enough a "technical term". The canonical reference is from CIDR https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf, but we have our own version from VLDB https://arxiv.org/pdf/2308.05368