szarnyasg

Joined 6 years ago

Karma

304

Submissions

Member since

May 23

Developer relations at DuckDB Labs, benchmark lead at GDC

View on Hacker News

Recent submissions

Comment · 5 months ago

Hello, DuckDB devrel here. First, thanks for the kind words :)

Second, it's funny you should mention the 20-line awk script. I was making a very similar argument yesterday at the Ubuntu Summit: at some point, using shell scripts with GNU coreutilus becomes impractical, while DuckDB SQL scripts scale better in terms of complexity and maintainability (and often also performance). My slides are here: https://blobs.duckdb.org/slides/duckdb-ubuntu-summit-2026.pd... (pages 32 to 36)

Third, MotherDuck develops a closed-source DBaaS on DuckDB. They build on DuckDB, and you connect to MotherDuck with DuckDB but they are a separate VC-funded company headquartered in Seattle. DuckDB is developed by DuckLabs, a bootstrapped (revenue-funded) company in Amsterdam. And the IP of the project is in a third organization: a Dutch non-profit called the DuckDB Foundation. For details, see https://duckdb.org/faq#how-are-duckdb-the-duckdb-foundation-...

Comment · 5 months ago

Yes, Quack resolves this problem. In particular, your client (likely a DuckDB instance) will talk to a remote DuckDB that both has access to the underlying storage and can also serve as the catalog itself.

Comment · 5 months ago

Well, we are really working on it: https://github.com/duckdb/ducklake/pull/1151

So you'll be able to test it in a few days.

Comment · 5 months ago

Hello, DuckDB DevRel here. Quack is independent from MotherDuck. MotherDuck has its own proprietary protocol, which has been around for years and it supports things like dual execution – see more here:

https://duckdb.org/quack/faq#what-is-the-relationship-betwee...

Of course, in the future MotherDuck can also support Quack, but this is not the only interesting use case for Quack.

DuckDB-Delta Grows Up: Writes, Unity Catalog and Time Travel duckdb.org

3 szarnyasg 5 months ago 0 comments

Comment · 6 months ago

DuckDB devrel here. You are right. This was in the FAQ but I also added it to the DuckLake documentation's main page at https://ducklake.select/docs/stable/

Comment · 6 months ago

DuckLake does not use B+ trees and it handles fragmentation with techniques like partial files and compaction upon checkpointing. The developers of DuckLake talks about this here: https://youtu.be/7Su0aVzbb-U?t=689

(Disclaimer: I work at DuckDB Labs)

Comment · 6 months ago

Hi, DuckDB DevRel here. To have concurrent read-write access to a database, you can use our DuckLake lakehouse format and coordinate concurrent access through a shared Postgres catalog. We released v1.0 yesterday: https://ducklake.select/2026/04/13/ducklake-10/

I updated your reference [0] with this information.

Comment · 7 months ago

That's a difficult question and I would like to avoid giving a direct answer (because I co-lead a nonprofit benchmarking graph databases) but even knowing what you need for a graph database can be a tricky decision. See my FOSDEM 2025 talk, where I tried to make sense of the field:

https://archive.fosdem.org/2025/schedule/event/fosdem-2025-5...

Comment · 7 months ago

That's a good point. I re-ran the benchmark on two instances:

- c8gd.4xlarge - this has a single 950 GB NVMe SSD.

- c5ad.4xlarge - this has 2 x 300 GB disks, which I put in a RAID 0 array. There are no c6ad.4xlarge instances, so this is the closes NVMe-enabled approximate to ClickBench's most popular choice, c6a.4xlarge.

I also added results from my local dev machine, a MacBook M1 Max with 64 GB RAM and 10 cores.

Here are the results:

  | machine        | cold_run_avg | cold_run_sum | hot_run_avg | hot_run_sum |
  | -------------- | -----------: | -----------: | ----------: | ----------: |
  | macbook m1 max |         0.48 |        20.68 |        0.43 |       18.60 |
  | macbook neo    |         1.39 |        59.73 |        1.26 |       54.27 |
  | c8gd.4xlarge   |         0.51 |        22.04 |        0.24 |       10.36 |
  | c5ad.4xlarge   |         1.29 |        54.14 |        0.55 |       22.91 |
  | c6a.4xlarge    |         3.37 |       145.08 |        1.11 |       47.86 |
  | c8g.metal-48xl |         3.95 |       169.67 |        0.10 |        4.35 |

On the cold run, the MacBook is on par with the c5ad.4xlarge. The c8gd.4xlarge is about ~2.5x faster on the cold run.

I know this is moving the goalpost, however, it's quite interesting that both of these cloud instances with instance-attached storage are still outperformed by the M1 Max (which is 4+ years old) on the cold run. And they would quite likely lose against the latest MacBook Pro with the M5 Pro/Max on both the cold and the hot runs. But that's an experiment for another day.

Comment · 7 months ago

Indeed, it would have been interesting but I really wanted to get the blog post out on the launch day of the MacBook Neo and did not have the bandwidth to run additional cloud experiments.

I ran TPC-DS SF300 now on the c6a.4xlarge. It turns out that it's still quite limited by the EBS disk's IO: while 32 GB memory is much more than 8 GB, DuckDB needs to spill to disk a lot and this shows on the runtimes. Running all 99 queries took 37 minutes, so about half of the MacBook's 79 minutes.

> Command being timed: "duckdb tpcds-sf300.db -f bench.sql"

> Percent of CPU this job got: 250%

> Elapsed (wall clock) time (h:mm:ss or m:ss): 37:00.96

> Maximum resident set size (kbytes): 25559652

Comment · 7 months ago

You're right! I pushed an updated TL;DR block.