zeroimpl
Joined 8 years ago
Karma
621
Submissions
450
Member since
Jan 11
Joined 8 years ago
- trim off all trailing delimiters: },"
- then add on a fixed suffix: "]}
- then try parsing as a standard json. Ignore results if fails to parse.
This works since the schema I’m parsing had a fairly simple structure where everything of interest was at a specific depth in the hierarchy and values were all strings.
But in general we aren’t trying to parse arbitrary documents, we are trying to parse a document with a somewhat-known schema. In this sense, we can parse them so long as the input matches the schema we implicitly assumed.
To me though the idea that to read like a single 16MB chunk you need to actually read like 4MB of data from 5 different hard drives and that this is faster is baffling.