aboodman
Joined 18 years ago
Joined 18 years ago
I don't see the gotcha here. I don't care about the offline part. I mean I accept that some people do, but that's not where the value comes from for many major synced products like Linear, Notion, Superhuman.
> complicated/over engineered cache
Sync engines are nothing at all like a cache - that's the point.
They are a replica. Caches are by nature inconsistent. Every entry in the cache is from a different point in time.
Sync engines replicate a consistent subset of your database to the client as an atomic unit. This enables things caches can't do:
* New, fresh queries can be returned instantly from local state
* Mutations can apply locally (optimistically) automatically, without custom code
* The UI updates automatically to reflect server changes
Whether these things are necessary depends on your application. But basically all productivity applications of any complexity keep re-implementing sync engines by hand so I guess most apps do in fact find them necessary.Whether sync engines generalize is an open question. Is it hard? Yes. Is it a distributed systems problem? Yes! Is it worth doing? I think it is. Web applications often suck and sync engines are an important part of many of the ones that don't. I want to enable that experience for more apps without teams having to build it themselves.
These systems are designed for the very common case of a global user base. If you have geographically centralized users, you can maybe do something simpler. That is rare in my experience - basically all of our customers have users worldwide. They typically don’t even know where their users are so making ux tradeoffs based on that feels really risky.
But maybe my experience is different than yours. One of the amazing things about the software ecosystem is how big it is. Everyone thinks their view is the common case.
https://zero.rocicorp.dev/docs/connection
Errors and connection are handled in a centralized place so that they automatically get applied to all paths.
Errors immediately disconnect the app and trigger UI. Writes are no longer accepted. After 1 minute of of failed connection attempts, same happens.
This formalizes and enforces the common pattern in popular sync-based apps of detecting disconnects fairly aggressively and warning the user.
But I get it. Unfortunately something like this cannot be a sidecar, or at least I do not know how to make it one. It's central to your app in the way that React is.
Fortunately the category is expanding and there are several sync engines that plug in exactly how Zero does - by replicating your database. You can switch between them easily. So as long as you think you do need a sync engine you aren't that married to Zero specifically.
Here's a demo of that!
Again, AWS latency us-west-1 to us-east-1 is 70ms. That's absolute best case for one round-trip that does absolutely no work. And it's ignoring the case of anyone outside of continental US.
Add in actual server-side work, db interactions, and contention - and you're quickly looking at hundreds of ms.
As for ZQL:
a) basically all of our customers already use Drizzle/Prisma. So they are very used to custom DSLs, and like them. I know, I was surprised to!
b) You typically use the same code client-side and server-side. There's no branching. The example you pasted is showing an escape hatch for when you want to use custom SQL. The option is there, but it's not the common experience.
This is what a typical mutator looks like:
```ts
// src/mutators.ts
import {defineMutators, defineMutator} from '@rocicorp/zero'
import {z} from 'zod'
export const mutators = defineMutators({
updateIssue: defineMutator(
z.object({
id: z.string(),
title: z.string()
}),
async ({tx, args: {id, title}}) => {
if (title.length > 100) {
throw new Error(`Title is too long`)
}
await tx.mutate.issue.update({
id,
title
})
}
)
})
```We are trying to make apps like Notion, Linear, Superhuman easier to create. These apps all uses custom-built sync engines that took their teams many person-years of effort to construct.
Whether this complexity is worth it depends on how badly you want instantaneous response. If you do, you will end up using sync one way or other, and you will end up with something roughly like Zero mutators.
Live demo: https://gigabugs.rocicorp.dev/.
We also list some alternatives here: https://zero.rocicorp.dev/docs/when-to-use#alternatives.
If you're interested in how these things work internally, check out the Replicache design doc: https://doc.replicache.dev/concepts/how-it-works. Replicache was the predecessor to Zero and the core protocol still works the same way.
Make a mutation to what?
The classic server rendered web-app doesn't have any data to make a mutation to. You could try to patch the UI but that would be a huge pita and not really a scalable (in effort) solution.
If you have an SPA, you still don't really have data on the client-side. You have a bunch of cached query responses. You can update those, but (a) it will be a pita to do correctly, (b) you'll have to do it to every possibly affected query, and (c) you have to remember to undo it at the right time (way more subtle than it appears - think it through!).
A sync engine creates the client-side normalized datastore that allows you to "do a mutation client side". In fact, you're kind of right that once you have a sync engine, just doing a mutation is really easy. The real challenge is all the infra required to enable you to do so.
What are you talking about? The only AWS region < 10ms away from us-east-1 is, err, us-east-2:
us-west-1 is 60ms away. eu-centra-1 is 100ms away. asia is 200ms away.
and this is datacenter-to-datacenter traffic. Actual latency over the public internet to residential providers is far worse.
Your database needs to be in exactly one region. So no matter where you put it, the majority of uses on earth are going to be > 100ms away from it.
It doesn't matter where the endpoints are, because the endpoints need to talk to the database to read and write data. Thinking you can replicate some data closer to the users? Congrats you are now the proud owner of a "local-first syncing" database. This replicated database you use (either of your own design or off the shelf) will have all the same problems of client-side syncing. Except you'll still have significant network latency.
There is no getting around physics. You either have quarter-second commits for most users or eventual consistency (aka syncing). Those are the options.
People management is about managing the company's resources to achieve goals. If you are not the one leading the implementation of those goals, you are not going to be able to:
* reason about what the right about of resources should be
* see opportunities for optimization
* forecast future need
You will be completely dependent on a technical lead who does have that information. So then what is your independent role? Just to shuttle information between the technical lead and others?I got to know him much better through these productive interactions then awkward smalltalk in a 1:1.
And it kind of make sense to meet privately quarterly since perf reviews are also quarterly and that's the only reason I can really think of for a private scheduled face-to-face.
Of course I could always just ask for a private meeting anytime I wanted, which I guess I did from time to time. But it always for a product reason: a tough tech choice I was wrestling with or similar.