exergy
Joined 3 years ago
Karma
9
Submissions
4
Member since
Sep 28
Recent submissions
Comment · 1 year ago
Comment · 2 years ago
there is Ibis[0] as a fairly mature package. They recently adopted duckdb as the default execution engine and it can give you a nice python dataframe API ontop of duckdb, with hot-swappability towards heavier engines.
With tools like this providing a comprehensive python API and the ability to always fall back to raw SQL, i am not sure DuckDB devs should focus on the python API at all beyond basic (to_table, from_table) features.
Impressive progress and a real chance to shake up the data tool market, but still a way to go: There is is still much to do especially on large table formats (iceberg/delta) and memory management when running on bigger boxes on cloud. Eg the elusive "Failed to allocate ..." bug[1] is an inhibitor to the claim that big data is dead[2]. As it is, we tried and abandoned DuckDB as a cheaper replacement for some databricks batch jobs.
[0] https://github.com/ibis-project/ibis [1] https://github.com/duckdb/duckdb/issues/12667, https://github.com/duckdb/duckdb/issues/9880, https://github.com/duckdb/duckdb/issues/12528 [2] https://motherduck.com/blog/big-data-is-dead/
Comment · 2 years ago
A (unfortunately still valid) running joke about German public administration is that November is the high season for fax machine sales people
Comment · 3 years ago
We use duckdb-wasm as an in-browser telemetry store for an IoT use case. beats the performance of other data stores in your browser and much simpler than manually wrangling arrays
DuckDB also has potential for IoT/embedded use cases as a data buffer with its compression and efficient reads, instead of just sqlite. but for that it needs to free up disk after expiring data
I thought AWS was dropping the ball on Open Table Formats, but this one is a smart move and a direct shot fired at Databricks with Delta.
Simply by making it the default for software engineers wanting a big data table on S3