free-ekanayaka

It's not a buzz term. It's really fully async disk I/O. Dqlite does not use SQLite's stock vfs implementation for writing to disk, as it's an entirely different model (based on raft).

Everything you said is very accurate (dqlite author here).

Give us time, we'll get there :)

Depends on how fast is your disk, what file system and kernel you use, and how low is your network latency. Difficult to predict. But it's basically as fast as it can get given 1) hardware constraints 2) raft consensus.

If you want light-speed insert/delete, you could probably don't use the disk at all: as long as a majority of your nodes don't die, you won't lose any data. You can also go somewhere in between and save to disk only at specific intervals.

Not that I know, but it's on my todo list (dqlite author here).

The support of async disk I/O in Linux differs depending on kernel version and file system type. But it is possible to get 100% async I/O with the is_submit(), and dqlite will leverage that if detected.

There is now a new async I/O API available in Linux (I'm not remembering the name right now, but it was developed by folks at Facebook). It looks promising so I'll check it at some point. (dqlite author here)

Oh I had no idea somebody was using it in the wild! It has been unstable until now, we just released v1.0.0 yesterday. So no more public API breakage from now on.

If you read the raft paper, or just watch presentations of it, you'll see that all the issues you mentioned are covered:

- faulty node replacement: you can take any node off (or any node can crash) at any time. As long as there are enough nodes left to reach a quorum, your system will be available. If there are not enough nodes left, your system will be unavailable (but keep consistency)

- re-syncs, snapshots and the rest are all covered in the raft paper

- clients wanting to perform writes always talk to the node that is currently the leader, that node fails, clients will look for next leader (which will be eventually elected as long as there's a quorum of surviving nodes)

- overwhelming a system is a different concern, raft writes are serialized so the goal is usually not throughput (for that you might look at AP/AC storage solutions in the CAP spectrum, raft is CP). Designs that need high write throughput might still use raft as internal building block for coordination.

"slower" means this:

when you perform a SQLite write, SQLite writes a new page on disk to the its write-ahead log. dqlite needs to replicate that page write across all nodes. A page is typically 4kb, so when you do a write on the leader node you need to transfer 4kb across the wire to all other follower nodes and wait for a quorum of them to come back and say "I got it" (which includes writing the page to disk). On the contrary rqlite just needs to transfer and store the SQL text, which is typically less than 4kb.

In practice it's still pretty performant, but I'll publish benchmarks later on.

I plan to submit the patch to SQLite upstream too, yes.

Unit and fuzzy-based test coverage that includes all possible edge cases is being put in place. I'd like to write a Jepsen test suite too.

Hardware does not matter that much, since SQLite is pretty hardened for handling any possible hardware failure (out of memory, out of disk space, memory/disk corruption) and propagate that to client code (including dqlite). Regarding network failures (hardware-related or not), raft guards you against them, and it's formally proved.