geoHeil
Joined 4 years ago
Karma
56
Submissions
24
Member since
Feb 16
Recent submissions
Comment · 3 months ago
I maintain a similar library more focused on data engineering needs: https://docs.metaxy.io/ maybe it is useful for more people.
Comment · 1 year ago
Benchmarks are like single match scorelines: interesting, but rarely comparable. We built a public ELO score leaderboard for analytics engines that treats every public benchmark (TPC-DS, TPC-H, SSB, vendor blogs, community posts - even the messy ones) as a “match.”
Why ELO? Because comparing databases is more like ranking football clubs over a season than timing a single sprint. Creating a relative robust and relative score is what counts.
Comment · 1 year ago
We are currently preparing a submission for JOSE and are looking for feedback but also more people who are interested in joining in and refining the template
Comment · 1 year ago
Together with www.linkedin.com/in/aaron-cul... I have created a template for making LLMs (from different vendors and even self hosted ones) easily accessible to researchers - including advanced document RAG with docling.
Comment · 1 year ago
Nice. What about tracking the learning process and comparing against the peer group/ class?
Comment · 1 year ago
As some others already posted: There is more to copying data than just moving it. It is about observability. A lot of companies have created their own frameworks.
One example I find useful for a lot of usecases is dagster. You can define resources to encapsulate complexity https://docs.dagster.io/guides/build/external-resources in fact with components build on top of custom DSLs https://docs.dagster.io/guides/labs/components
At Magenta/Telekom we are bulding on https://georgheiler.com/event/magenta-data-architecture-25/ - you can follow along with this template here https://github.com/l-mds/local-data-stack/ you may find these examples useful to understand how to use dagster/graph-based data pipeliens @scale
Comment · 2 years ago
Did you ever want to find an alternative to cloud data PaaS/SaaS vendor lock-in?
In my blog post, I show how a 0-cost switch between local, Databricks and AWS EMR environments can be achieved to save cost by using dagster. Additionally, software engineering best practices around testability, and producing DRY code are enhanced.
Check out my deep dive and let’s discuss the potential impacts on our projects!
Big thanks also to my co-author Hernan.
Comment · 2 years ago
Interesting why? What were the pain points?
Comment · 2 years ago
If you use bigquery in big lake mode - i.e. with parquet or iceberg or delta files on GCS (or other object store) you can easily pull in the data into duckdb as well