philbe77
Joined 1 year ago
Karma
80
Submissions
44
Member since
Mar 20
Recent submissions
Comment · 5 months ago
TypeScript: https://www.npmjs.com/package/@quack-protocol/sdk
Comment · 5 months ago
Two new Quack client drivers:
ADBC: https://github.com/gizmodata/adbc-driver-quack JDBC: https://github.com/gizmodata/quack-jdbc
Comment · 8 months ago
Now it is easy to use Power BI with a DuckDB server you can self-host: GizmoSQL. Powered by Arrow Flight SQL - you can run terabyte-scale analytics for dirt-cheap...
Comment · 11 months ago
Oh, and GizmoData Cloud (SaaS option) is coming soon - to make it easier than ever to provision GizmoSQL instances...
Comment · 11 months ago
GizmoSQL is definitely a good option. I work at GizmoData and maintain GizmoSQL. It is an Arrow Flight SQL server with DuckDB as a back-end SQL execution engine. It can support independent thread-safe concurrent sessions, has robust security, logging, token-based authentication, and more.
It also has a growing list of adapters - including: ODBC, JDBC, ADBC, dbt, SQLAlchemy, Metabase, Apache Superset and more.
We also just introduced a PySpark drop-in adapter - letting you run your Python Spark Dataframe workloads with GizmoSQL - for dramatic savings compared to Databricks for sub-5TB workloads.
Check it out at: https://gizmodata.com/gizmosql
Comment · 1 year ago
good point :) - we can re-aggregate HyperLogLog (HLL) sketches to get a pretty accurate NDV (Count Distinct) - see Query.farm's DataSketches DuckDB extension here: https://github.com/Query-farm/datasketches
We also have Bitmap aggregation capabilities for exact count distinct - something I worked with Oracle, Snowflake, Databricks, and DuckDB labs on implementing. It isn't as fast as HLL - but it is 100% accurate...
Comment · 1 year ago
hi sdairs, we did store the data on the worker nodes for the challenge, but not in memory. We wrote the data to the local NVMe SSD storage on the node. Linux may cache the filesystem data, but we didn't load the data directly into memory. We like to preserve the memory for aggregations, joins, etc. as much as possible...
It is true you would need to run the instance(s) 24/7 to get the performance all day, the startup time over a couple minutes is not ideal. We have a lot of work to do on the engine, but it has been a fun learning experience...
Comment · 1 year ago
This is something we are trying to take a novel approach to as well. We have a video demonstrating some TPC-H SF10TB queries which perform inner joins, etc. - with GizmoEdge as well: https://www.youtube.com/watch?v=hlSx0E2jGMU
Comment · 1 year ago
I've since learned (from a DuckDB blog) - that DuckDB seems to do better when the XFS filesytem. I used ext4 for this, so I may be able to get another 10 to 15% (maybe!).
DuckDB blog: https://duckdb.org/2025/10/09/benchmark-results-14-lts