Comment by nijave

I would think there has to be some decent middle ground like storing some structures on disk mmap'd and letting the kernel handle back pressure and caching