creationix

I see. We have different definitions of serialized. The way I typically see it used is to describe how something is transmitted and stored, not how it is read.

Yes, by your definition, this is random and not serially read.

So you're saying that random access formats that are encoded to disk as a stream of bytes are not "serialized" because you don't alway read them in order?

Yes, many formats are read start-to-end, but I don't think that's a requirement. The important thing is it can be stored and transmitted as a stream of bytes. The word describes how it is transported and stored, not how it is read.

> Just copy and paste files

If all your workflows allow copying as binary files, more power to you! But there are a lot of workflows where that is not possible. This was inspired by years of hands-on operational incident handling in production systems. Every time we use a binary format, it's extra painful.

This particular format would be slightly more compact as binary, but not enough to justify closing the door on all the use cases that would preclude.

I'll probably add a binary variant for people who prefer that (or for people who want to be able to embed binary values in the data without base64 encoding it)

> what sort of storage device does not allow your computers to use all 256 byte values

- clipboards

- logs

- terminal output

- alerts

- yaml configs

- JSON configs

- hacker news comments

- markdown documentation

- etc...

I assure you, this is not a solution looking for a problem. I started out with binary encodings first, but then realized it limits so many workflows.

And don't worry about railroad diagrams. I already intended to create them, I've just been extra busy this week with other things.

Tha main reason for the reverse encoding is it makes it easier on the writer. You simply do a depth-first traversal of the data graph and emit data on the way back up the stack. Zero buffering is needed since this naturally means you write contents before the length prefix.

But it does open up a future direction I want to make with mutable datasets using append-only persistent data structures. The chain primitive is currently only used for strings, but it will be used to do the equivalent of `{...oldObj, ...newObj}` as a single chain `(pointerToOldObj, newObj)`.

With chains and pointers, you can write new versions of a dataset and reuse all the existing values that are unchanged. This, combined with random-access reads and fixed-block caching makes for a fairly complete MVCC database.

Also good luck parsing 10 MiB of JSON in a loop that can't tolerate blocking the CPU for more than 10ms.

What's expensive is very relative to the use case.

Very true. I had forgotten about bencode, I should read up on that again.

It makes sense they need a canonical form because they want same values to have same content hashes.

I meant computers can read it without any preprocessing. It's random access. You don't need to parse it, you don't need to decompress it. You just start at the end and follow pointers till you get to the desired value.

Even a trivial doc like this is challenging for me to read as a human.

yeah, LuaJIT is one of the use cases I had in mind working on this. JSON is pretty fast in modern JS engines, but in Lua land, JSON kinda sucks and doesn't really match the language without using virtual tables.

JSON has `null` values with string keyds, but lua doesn't have `null`. It has `nil`, but you can't have a key with a nil value. Setting nil deletes the key

Lua tables are unordered. But JS and JSON are often ordered and order often matters.

RX, however matches Lua/LuaJIT extremely well and should out-perform the JS Proxy based decoder using metatables. Since it's using metatables anyway do to the lazy parsing, it's trivial to do things like preserve order when calling `pairs` and `ipairs` and even including keys with associated null values.

You can round trip safely in Lua, which is not easy with most JSON implementations.

> it only has a text encoding as long as you can guarantee you don't have any unicode?

The format is technically a binary format in that length prefixes are counts of bytes. But in practice it is a textual format since you can almost always copy-paste RX values from logs to chat messages to web forms without breaking it.

unciode doesn't break anything since strings are encoded as raw unicode with utf-8 byte length prefixes. It supports unicode perfectly.

If your data only contains 7-bit ASCII strings, the entire encoding is ASCII. If your data contains unicode, RX won't escape it, so the final encoding will contain unicode as UTF-8.

How does CBOR retain JSON compatibility more than RX?

RX can represent any value JSON can represent. It doesn't even lose key order like some random-access formats do.

In fact, RX is closer to JSON than CBOR.

Take decimals as an example:

JSON numbers are arbitrary precision numbers written in decimal. This means it can technically represent any decimal number to full precision.

CBOR stores numbers as binary floats which are appriximations of decimal numbers. This is why they needed to add Decimal Fractions (Tag 4)

RX already stores as decimal base and decimal power of 10. So out of the box, it matches JSON