ongardie

The issue if realms stored HMAC(realm_id + PIN), where PINs are presumed to be low-entropy, then an individual realm could brute-force the PIN. Specifically, an adversary with access to a single realm's database could enumerate PINs, run them through the HMAC along with the realm ID, and test locally whether that's the correct PIN. That would already be bad because users might reuse PINs across services. Then, if the adversary had valid auth tokens for other realms, they would be able to use that PIN to recover the secret shares from other realms and reconstruct the secret.

The Juicebox protocol is designed to prevent this. A realm can't individually test whether or not a PIN is correct.

Note: I'm a former employee of/contributor to Juicebox.

If the network is split into AB and CDE, then only CDE will be able to reach a quorum. If a client is on the AB side of that split, no consensus algorithm could guarantee freshness on a read. LogCabin clients will continually retry until they can talk to a functioning leader (and clients can put a timeout on that) so that you get a guaranteed fresh read every time; other systems could choose to return a stale result right away instead. It's up to the implementation, but I strongly recommend people not do stale reads by default.

Yes, good point. I welcome Mr. McCaffrey to give it a spin, if he's so willing. And though there may be critical bugs left to find, I don't feel like I've misrepresented the current state of LogCabin.

Good timing then, and I'd be happy to talk more about whether LogCabin is a good fit for your use.

RAMCloud used to depend on an earlier version of LogCabin (before the data model was a key-value tree), then John rewrote the RAMCloud end of that and switched it to using ZooKeeper, but he made it pluggable on the RAMCloud side. So we just need an "ExternalStorage" implementation for LogCabin in RAMCloud to make that work again. It should be relatively straightforward and I'm confident they'd welcome a well-written and tested patch, but no one has volunteered yet.

I guess the answer is no, I haven't really looked at it. Anyone have experience with it in a large-ish project outside of Node?

Maybe, but I don't think it matters much. The comment there explains the issue, and I vaguely remember my measurements showing that this wasn't a big deal.

I do think it's an interesting (dare I say) limitation of epoll, and something I'd think about if I was redesigning epoll.

Please file an issue on github if you have ideas on how to improve this without adding significant complexity.

In part that's because I don't see other Raft implementations as competitors. A major goal in Raft was to enable many implementations. It'd be a huge fail if everyone switched to LogCabin and abandoned all the other Raft implementations.

Really though, it's probably not a feature list that'll make people want to use LogCabin. In Scale Computing's case, it's that they have a C++ code base that they could integrate LogCabin well with, and they know they can work with and maintain the code if they need to.

This question came up at the CoreOS Fest earlier today too. I think everyone wants libraries as well as services, and I'd like that too for LogCabin one day. It's just a bit harder when you start thinking about the details. It's not impossible though. As one example, CockroachDB is using etcd's Raft implementation as a library.

Part of the issue is that we want to have libraries with small simple interfaces, and a consensus library is probably going to be on the large side, as it has to interface with the disk, the network, and the state machine, plus membership changes, log compaction, who's the leader, configuration settings, debug logging, etc.

Another issue is that, as a library, it has to be in the language you're using. And if that language is C++ or similar, it has to be compatible/convenient with whatever threading approach you're using.

Then there's performance/memory. Some Raft implementations are designed to keep relatively small amounts of data in memory (like LogCabin or etcd), and others are meant to store large amounts on disk (like HydraBase). Some are optimized more for performance, others for safety.

I think we'll get libraries eventually. Keep in mind Raft is still very young. Paxos is around 26 years old, Raft is only .5 to 3 years old (depending on when you start counting). I like to think that Raft lowered the cost of developing a consensus system/library significantly, but it still takes time to develop mature implementations. Right now we have a lot of implementations in early stages; I wonder if some of these efforts will consolidate over time into really flexible libraries.

There's not currently an extern "C" wrapper, but I don't think that'd be too much work to write.