Comment

Comment by fnord123

According to this Data Engineering interview (beginning around 8m in) the improvements begin with moving from the JVM which has a latency impact. But the real improvement is that fsync on a file handle puts in a barrier in the FS. RedPanda does some work around batch coalescing, they skip the page cache to side step a bunch of kernel locks, and adaptive allocation (preallocating FS space):

https://vectorized.io/dataengineering-podcast-interview/

The throughput is 4x, the latency is 10x.

parent fnord123 6 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton