Apologies for being a debbie downer, but it looks like the authors looked into Protocol Buffers and Flatbuffers.
Did they consider CapnProto?
In my limited experience with efficient serdes for analytics, CapnProto would meet 1 - 4.
Arrow is an in-memory format and CapnProto would afford the ondisc (memmap) heavy lifting with (very) little overhead. My
CapnProto/LMDB datastructures allow me to write better than C++ code in Python with comparable runtime performance.
The only issue I have with CapnProto is getting it to run on Windows 7 x64, but it works out of the box on Linux.
Did they consider CapnProto?
In my limited experience with efficient serdes for analytics, CapnProto would meet 1 - 4.
Arrow is an in-memory format and CapnProto would afford the ondisc (memmap) heavy lifting with (very) little overhead. My CapnProto/LMDB datastructures allow me to write better than C++ code in Python with comparable runtime performance.
The only issue I have with CapnProto is getting it to run on Windows 7 x64, but it works out of the box on Linux.