Comment

Comment by kentonv

That's incorrect. When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing".

Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able accessor functions to manipulate these structures which do pointer arithmetic similar to what a compiler would generate for accessing a struct. The end result is that accessing primitive fields from a Cap'n Proto struct is essentially identical in terms of machine instructions to accessing fields of a C struct.

To call that "deferring the parsing to access time" does not make sense.

(Note that for pointers, there are more differences: namely, because data will not always be loaded at the same address, pointers need to be relative rather than absolute. They also need to be bounds-checked for security. This adds a few instructions to pointer accesses, but those instructions look nothing like traditional "parsing" and certainly aren't byte-by-byte operations.)

Your statement earlier:

> The correct way to handle binary data is to unpack it into the struct byte by byte.

This is inaccurate. Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid. It is, for example, usually very hard to do "random access" with a byte-by-byte parser, because allowing out-of-order parsing tends to blow the code complexity through the roof.

On the other hand, with Cap'n Proto and similar approaches, you can trivially mmap() a very large data structure and traverse it randomly, and it "just works".

(Disclosure: I'm the author of Cap'n Proto, as well as the author of the first open source release of Google's Protocol Buffers, which does byte-by-byte binary parsing.)

parent kentonv 8 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton