elendilm

Reducing attack surface is a lesson for everyone. Newcomers are impressed by massive piles of code.

Seasoned developers are unimpressed, or even disgusted, by the same massive piles of code.

I first encountered this behaviour on Linux desktop with Qt QML apps as far back as 2015. For a long time, I thought it was Qt QML ListView's inefficiency and didn't think much of it.

But in recent years, the phenomenon was visible with Dip which was built on rust.

Dip does massive parallel fanout using green threads ('may' coroutines) with work being NUMA pinned (software hash) using a thread per core model.

A default request may span up to _chunk_count_ jobs (default 5k) configurable per request.

Every request (x 5k jobs) from the client on a stress test sees a jump of ~10-20 MB of memory. Sometimes the jumps use large swaths of memory - say 100s of MB and stabilizes for a while only to gradually climb up again later. Rinse and repeat - OOM kicks in and the process crashes.

Many times we thought we were leaking memory somewhere. But armed with the Qt experience a decade ago, we intuited maybe its the allocator - and switched to mimalloc on desktop executable.

Dip's memory, using mimalloc, stabilizes at ~60 MB post request completion no matter how many requests you throw at it.

Pity that on mobiles, more specifically Android, we have to live with the default allocator on even trivial Qt apps let alone embedded Dip. But people have lived with it for at least a decade, so it is what it is.

Thus on mobiles, we just have to be content like everybody else, unable to squeeze every last ounce of performance and efficiency out of the hardware on stress tests ensuring Dip remains "just better :(" :)

True. The guide is an interesting read. When you are deep in the trenches, most of what the github link says is just common sense.

Eg: < Optimize for the slowest resources first (network, disk, memory, CPU) in that order, after compensating for the frequency of usage, because faster resources may be used many times more. For example, a memory cache miss may be as expensive as a disk fsync, if it happens many times more.>

This is just obvious common sense. Great read.

mimalloc aggressively reclaims space and it benefits the use case in our Dip database built on rust.

This is for a long running process where consuming compute and memory should only occur on processing a request and must revert to a baseline memory consumption once the request is completed.

malloc with default allocator always gradually increases the memory consumption. This continues with enough request bursts on our testing until the process runs out of memory. mimalloc doesn't.

On mobile, we are stuck with the default allocator. On mobiles, the phenomenon of the default allocator consuming memory is reproducible even with normal Qt apps. It is what it is.

Most still get benefits. I concede that many don't get "direct" remuneration. Some journals do pay. But many do get vouchers, discounts, free access, early access, etc. which amounts to a form of payment and thus compensated in kind. This effectively is "paid" as far as I am concerned.

The logic stands. A reviewer, even those who get zero compensation, can always review publicly published papers just like others.

What has that got to do with anything? Look into his work rather than his online persona.

It's his/her prerogative to keep an online footprint or not.

Peer review and scrutiny out in the public is more valuable than only done by a couple of paid guys from a journal.

It's fairly basic. For a work published on a public platform, the same guys paid by the journal can also review it if they so choose to in addition to those people in the public who were not paid by anyone.

This looks great.

We have smart batching in Dip where we deduplicate work received in what we call a "batched" request. We use BTreeMap to deduplicate work in the preprocessing "scatter" phase.

This deduplication in smart batching is the slowest phase on most bulk write workloads (200k+ inserts in a single request) and takes sometimes up to 3x the time taken for the combined fanout + db writes + db commits + gather.

Will try out with Bbtree and see how it goes.

You are asking all the right questions. The answer is obvious.

Many don't want to say it or admit it.

That literally was history in the making. Much respect.

> More CPUs pegged at 100% is always a good sign of smooth flows, but optimizing further requires data layout changes from that point.

True. This is true by experience. I want to add that, additionally this is not just true based on experience, rather it is fairly a straightforward logic.

If the cpu is already saturated at 100% doing work in the most "efficient" way possible for a given data structure (some sort of a shannon limit for the given data structure), then there is zero further improvement possible.

Any additional improvement must therefore happen either on the thing that the CPUs operate on (data structure improvements) or on the hardware themselves.

I love how you first confused application managed indexes with B+trees in code. Then you moved to how you could not see how "this could be faster", asserting there is an additional indirection layer. When pointed out that the indirection also exists in database managed indexes, you wanted examples of usage at scale with a blog post. You also mentioned you have operated 1M QPS, yet you couldn't fathom while being oblivious to lock contention. You were persistent in demanding one public link.

When the Uber blog link was provided with exact scenario in the excerpt, you pivoted to how their new version has schema enforcement. You claimed you were able to prove all inefficiencies you encountered were due to bad data modeling or bad decisions which went against RDBMS.

When pointed out that the newer blog post doesn't mention removing the secondary indexes which was the whole issue, you digressed to the grand finale of absurdly claiming Sqlite will saturate phone's I/O before one saturates Sqlite, which is so false that it is not even funny. And finally demanding our Dip's benchmarks (which will come in due time, if? we ever release it to the public).

All of this when more than enough hand-holding was made to help you understand. Even when the underlying physics was easy to understand. Enough pivoting.

Let our conversation stand and be of help to anyone curious enough to learn the tradeoffs discussed. Best wishes.

> Everywhere I have worked that has DBs they claim are too large, I have been able to prove (typically by cloning one and remodeling it) that the reason they're suffering from performance issues is because they've made a series of decisions which were fighting against their RDBMS

Some are just physics. Say if you want SSD write saturation and multiple parallell write transactions, the traditional RDBMS model itself may not suffice. Its not just data size but concurrent write transactions, lock contention, etc. Additionally, think resource constrained devices like mobile phones. Or an aggressive cost reduction mentality where each cpu cycle shouldn't be unnecessarily wasted without feeling guilty. If you come from a systems engineering background like that, you wouldn't be much impressed with many traditional databases at all. I am generally not much interested in SQL databases anyway.

> They needed to create secondary indices as tables because they were storing everything in JSON blobs, which is difficult to directly index

Our database Dip stores raw binary as well as json as first class citizens. Can't speak for Docstore, but as far as I see, it is just an evolved form of its earlier self with different tradeoff according to their workload. Note the lack of mention of the removal of secondary indexes as separate data.

> I would counter that if you keep needing schema changes, you haven't done a good job modeling.

Disagree. Schema stabilizes - yes. But doesn't mean they don't go through cycles of change occasionally.

> Distributed querying may still just be a workaround rather than something anyone actualy wants to do as a first option.

That is not the case. Even on a single machine with multiple shards (think hundreds or even thousands of kv engine say LMDB envs), a distributed query parallelizes efficiently. Additionally it can also saturate the SSD/NVMe with multiple write transactions concurrently.

It definitely is the first option and not a workaround. Even replication benefits from this write saturation and reduced latency and parallelism. Naturally, multiple writes involving the same logical unit of work can go through a durable execution framework (think ordered-log/kafka) as one insert and the client dismissed only for the consumer later attempting multiple idempotent writes (logical work with retry on failure) - achieving maximum SSD saturation. The client is later informed of the status.

> Most of the big offline use cases I’ve seen are aerospace/drones or logistics or robotics ...

Our previous ERP (built almost over a decade) for a large enterprise client, required disbursing customers as the absolute paramount priority regardless of internet availability - which is where offline first became the only choice. They wanted data to seamlessly move across thousands of offline systems - which is where we wrestled with replication/conflict resolution, etc. Our new Slyp and SlypBusiness is a reincarnation/re-imagining of this old ERP with added goodies but as mobile first applications.

There is only up to a point where we tolerate other databases, and beyond which the conventional wisdom starts to be a net negative and we realized we are better off building one for ourselves.

> Okay, again, what kind of scale are we talking about here? 100ks of writes/second

You seem to be confusing writes/second with concurrent write transactions/second. Nvme won't rescue you if Postgres locks up only allowing each write txn to proceed sequentially when what you need is SSD write saturation on separate write transactions on multiple shards in a single machine.

> You'd be way better off going all NVMe today as your first step to "scaling" (in fact you are insane if you are using anything else for an OLTP database), and just that is likely to immediately solve whatever problem you have already.

This is hilarious. So we should all start building based on your prescription.

> what exactly is the benefit of manually implementing what it already does well?

Really? Is that a serious argument. Postgres is not a magic database that does everything that everybody wants very well.

In systems engineering, the trade offs are obvious. But if it is not clear at this point, I don't think I can offer much help.

I will offer one last advice.

Just write a snippet of code inserting 100M small keys and values into, say 1k chunks/shards of separate kv engine (LMDB, whatever). Do normal hashing for shard selection. Now store secondary indexes likewise. Now do parallel fanout for secondary indexes for selecting some 100k data. Collect the index data and do another parallel fanout and fetch the final data. Now go see the latency results (excluding lmdb env setup time). Additionally observe the latency for writes :-)

Rich index metadata is just icing on the cake. Cheers.