jdvh

We've spend the last couple of years building a local-first end-to-end encrypted multiplayer notes app (Thymer). We also explored running sqlite in the browser but ultimately concluded that we didn't really sqlite or a real database at all in the browser.

What does a document system need? A few simple indexes that track links and backlinks, mentions, and hashtags. You’re broadcasting change notifications anyway (for reactive DOM updates), so updating the indexes for those manually isn’t much extra work. You apply updates eagerly and then notify other users so they can apply the same updates, with some rules to guarantee that all users end up with the same state regardless of the order in which they receive the updates. But a relational database doesn’t help with any of this. Document systems tend to be versioned, so every user action turns into another entry in a transaction log. Even queries like “last Monday’s version of my document” don’t map naturally to SQL. You can query for transactions in a given time period, but unless you have snapshots, you’re still forced to re-apply transactions all the way from t=0 if you want to re-create the document state at a given date.

In theory you can have these "zero cost abstractions" but in practice I don't think so. The user manual gets thicker. Concepts like 'delete permanently' and backup/restore get more complicated. Users will get confronted by scary "advanced users only" warnings in the interface. Some enthusiast blogger or youtuber will create content highlighting those advanced features and then regular users will get themselves in trouble. Customer support gets way more complicated because you always have to consider the possibility that the user has (unknowingly) used these advanced features. If you put buttons in the interface users will press those buttons. That's just a fact of life. Advanced features always come at a cost. Sometimes that cost is worth it, but only sometimes.

Rule of thumb: every 10% increase in complexity cuts your potential user base in half.

This is why people make backups by copy-pasting files. This is why Excel is so dominant. This is why systems like hypercard and git are not mainstream and never will be.

There is a large universe of tools people would love if only they would bother to learn how they worked. If only. Most people will just stick to whatever tools they know.

For most people the ability to go back and forward in time (linear history) is something they grasp immediately. Being able to go back in time and make a copy also requires no explanation. But having a version tree, forking and merging, having to deal multiple timelines and the graphs that represent them -- that's where you lose people.

You're free to argue that this isn't "pure" CRDT, but the CRDT algorithm still runs normally, just a bit later than it otherwise would.

I don't think that's required, unless you definitionally believe otherwise.

When clients disagree about the the order of events and a conflict results then clients can be required to roll back (apply the inverse of each change) to the last point in time where all clients were in agreement about the world state. Then, all clients re-apply all changes in the new now-agreed-upon order. Now all changes have been applied and there is agreement about the world state and the process starts anew.

This way multiple clients can work offline for extended periods of time and then reconcile with other clients.

It's still early, but we have a checkpointing system that works very well for us. And once you have checkpoints you can start dropping inconsequential transactions in between checkpoints, which you're right, can be considered GC. However, checkpointing is desirable anyway otherwise new users have to replay the transaction log from T=0 when they join, and that's impractical.

As long as all clients agree on the order of CRDT operations then cycles are no problem. It's just an invalid transaction that can be dropped. Invalid or contradictory updates can always happen (regardless of sync mechanism) and the resolution is a UX issue. In some cases you might want to inform the user, in other cases the user can choose how to resolve the conflict, in other cases quiet failure is fine.

Completely decentralized is cool, but I think there are two key problems with it.

1) in a decentralized system who is responsible for backups? What happens when you restore from a backup?

2) in a decentralized system who sends push notifications and syncs with mobile devices?

I think that in an age of $5/mo cloud vms and free SSL having a single coordination server has all the advantages and none of the downsides.

You can think of ejectable as the opposite of being locked-in.

In exchange for perpetual development and maintenance costs. Total cost of ownership doesn't go down by rolling your own in-house.

Absolutely. There is no phone-home of any kind.

I think the main problem is lock-in. If you can't get your data out you can't leave. This is true for open source and for commercial products alike.

If you own your data and if you have the option to self-host you can always opt out of updates you don't like.

Certainly not. I don't believe progress is always good. But subsidies should be reserved for ambitious projects that push the state of the art forward. For those projects that realistically will not get funded commercially. CERN, for instance.