murtazayusuf

Thanks!

With 1.1 we are on par when compared only based on command execution, however it terms of IO kividb performs better. As you can see above, at pipeline 1, the performance is slightly worse than Dragonfly, but when pipelined at 16, it reached 21M. I don't have the flamegraphs with me from previous run, but given the similar ask from other hackers in this thread, I am going to release all of it asaic, I haven't yet because I didn't record them in my previous runs especially the 16 vs 1 pipeline numbers.

thanks for sharing your thoughts, you have the architecture right, kividb uses shared everything with fine-grained locks, similar to keydb and garrnet. it uses 8 shards per core, each with a hash map behind an rwlock, with separate locks for collection values. we lock all relevant shards in a fixed order. Multi and Exec and arbitrary-key scripts run behind a server-wide exclusive gate, so the transaction-heavy workloads serialize and also lua scripts with shared hash tags can run in parallel, which works well for use cases like bullmq. and for snapshots, we are avoiding forks and copy-on-write memory spikes by capturing immutable string references but the tradeoff is a write pause that grows with key count, which is not an issue with dargonfly for it's non-blocking snapshotting. Please do try this design with dragonfly as it is also a replacement to memcache which kividb isn't yet, haha

And thanks for vouching for me, not sure why it was marked as dead

a correction here, it wasn't the GC pauses within dragonfly itself, there were periodic dips in the throughput graphs overtime, that looked similar to GC pauses, but can't be the case as dragonfly does manual memory management. For the memory spikes around snapshots though, I certainly noticed them.

It is mostly because kividb can utilize all the cores you can allow on a machine, whereas Antirez built redis as a single threaded cache. The fairer comparison imo would be with redis cluster row in the table where 32 Redis shards were created, one per core, resulting 4.7M pipelined GETs/s. KiviDB’s 21.2M against that is down to not paying for cluster routing and per-shard overhead, and to io_uring batching.