chrisjc
Joined 8 years ago
Karma
1,262
Submissions
680
Member since
Mar 28
Recent submissions
Comment · 1 year ago
Comment · 1 year ago
duckdb -cmd 'CREATE TABLE my_data AS FROM READ_PARQUET($$data.parquet$$)' -ui
`duckdb -ui sqlitedb.db` should work bc duckdb can read sqlite files. If it doesn't autoload extension, you can INSTALL/LOAD in to your ~/.duckdbrc
Comment · 1 year ago
duckdb -ui pre_seeded_db.db
duckdb -ui -init startup.sql
Alternatively place statements/commands in `~/.duckdbrc` and just run `duckdb -iu`.
Comment · 1 year ago
> In the worst-case scenario, they would be 2-3 years behind the cutting edge, which is not mission-critical.
It's also worth considering how such an event might affect the US and allies. Would it slowdown, perhaps even halt certain operations/efforts for the US. For instance, all those chips the US needs to build supercomputers for "weather research". ;)
Comment · 1 year ago
Haven't tried it out yet, but the release notes look very promising esp relating to Arrow interactions
- Push dynamically generated join filters through UNION, UNNEST and AGGREGATE
- Fix arrow table filters
- [Arrow] Fix scan of an object providing the PyCapsuleInterface when projection pushdown is possible.
- DuckDB Arrow Non Canonical Extensions to use arrow.opaque
- Arrow Extension Type to be registered in DuckDB Extensions
- [Python] Use an ArrowQueryResult in FetchArrowTable when possible.
Comment · 1 year ago
Perhaps I don't fully understand what you're saying
CREATE TABLE person (name VARCHAR, age BIGINT);
INSERT INTO person VALUES ('Alice', 37), ('Ana', 35), ('Bob', 41), ('Bea', 25);
CREATE OR REPLACE MACRO my_query_table(t) AS TABLE (SELECT * FROM query_table(t));
SELECT * FROM my_query_table('person');
CREATE OR REPLACE MACRO query_macro(m, i) AS TABLE (SELECT * FROM query(CONCAT('SELECT * FROM ', m::STRING, $$($$, i::STRING, $$)$$)));
SELECT * FROM query_macro('my_query_table', 'person');
Comment · 1 year ago
I've been eagerly awaiting this for a couple of months now. And I've long wondered why there hasn't been such an implementation/extension for Flight especially as there are extensions for mysql, postgres, sqlite...
Only seems natural that an extension could developed for ATTACHing to a Flight server.
Looking forward to watching your talk.
Comment · 1 year ago
For read-oriented interactions...
ADBC --sql--> Flight SQL Server --flight--> Flight Servers (plural) --> datasources
SELECT * FROM <relation> WHERE ...
Remember that Flight/Flight-SQL is not a query engine replacement, but rather a way to communicate and exchange data between components with different responsibilities efficiently.
Comment · 1 year ago
Thank you for all the work you guys do. The Arrow ecosystem is just absolutely incredible.
My few gripes related to interop with duckdb are related to Arrow scanning/pushdowns. And this extends to interop with other projects like pyiceberg too.
Registering an Arrow Dataset (or pyiceberg scan) as a "duckdb relation" (virtual view) is still a little problematic. Querying these "relations" does not always result in an optimal outcome.
For Arrow datasets, you can intercept the duckdb pushdown, but duckdb will have already "optimized" the plan to its liking, and any scanning restrictions that may have been more advantageous based on the nuances of the dataset might have been lost. Eg:
WHERE A IN (3, 5, 7)
Perhaps in a similar way, turning an pyiceberg scan into a relation for duckdb effectively takes the entire scan and creates an Arrow Table rather than some kind of pushdown/"scan plan" for duckdb to potentially make more efficient with its READ_PARQUET() functionality.
Most of this is probably dependent on duckdb development, but all of the incredible interop work done across communities/ecosystems so far gives me a lot of confidence that these will soon be matters of the past.
Comment · 1 year ago
And all the Arrow parts work together quite nicely.
ADBC client --> Flight SQL (duckdb/whatever) --> Flight --> ?
It's quite an exciting space, and lots of projects popping up around Arrow Flight and duckdb.
Comment · 2 years ago
Since you're using python, have you looked into sqlglot? I think it has some pretty-print options.
Comment · 2 years ago
Maybe something like this?
https://duckdb.org/2024/10/02/pyodide.html
Comment · 2 years ago
Probably also worth mentioning that DuckDB can interact with SQLite dbs.
https://duckdb.org/docs/extensions/sqlite.html https://duckdb.org/docs/guides/database_integration/sqlite.h...
Thus potentially making duckdb an HTAP-like option.
Comment · 2 years ago
Probably not exactly what you mean or asking for, but the work Motherduck is doing looks promising.
https://motherduck.com/blog/differential-storage-building-bl...
Hopefully it finds its way into duckdb's repo some day.
Comment · 2 years ago
Ibis looks very promising.
There are also other ways to use both.
https://duckdb.org/docs/guides/python/polars.html
All of this dataframe compatibility is awesome. (much thanks to Arrow and others)
It should be extremely simple for databases that support ADBC (for example Snowflake, PostgreSQL).
For others it might just be a matter of mapping DDL, DML, DQL, etc to a supported database protocol driver (JDBC, ODBC, etc). Of course this is where things may get challenging as it would become the responsibility of your server to convert result to Arrows (tables/streams/etc). But could potentially be delegated to "worker" Flight servers (not a Flight SQL server) and then the server could return/forward their Arrow results (Flight results).
Of course some of this is to some degree already possible through DuckDB's MySQL/Postgres Extensions.
I imagine this could also be useful for developing/testing locally?
It might also provide a way to interchange databases while potentially easing database migrations (vendor to vendor) if ADBC isn't supported by the vendor.
Another potential value-addition could be to provide SQL dialect management by providing Substrait conversions (or sqlglot but looks like the server is Java, so unsure if possible, maybe Graal?).