klauspost

Available via cli and Go API

About the versioning, glad you spotted it anyway. There isn't as much use of the gzhttp package compared to the other ones, so the bar is a bit higher for that one.

Also making good progress on getting a slimmer version of zstd into the stdlib and improving the stdlib deflate.

The main reason I took the effort to write the spec was to encourage ports - and commit to the format.

I've converted a basic block encoder/decoder to C (see end of README for link), to assist anyone wanting to get started on a C implementation.

Having been 10+ years since I've done any serious C, I didn't want to assume I could write a good C API for it.

If anyone are interested in picking it up, reach out.

> comparisons with zstd & lz4

The repo has lz4 comparisons (Go version) for both blocks and streams.

I see MinLZ more like a compliment to zstd, just like lz4 - where speed matter more than ultimate size.

Here is the fully parallel speed of block decompressing with zstd or minlz - Go versions:

* Protobuf - zstd: 31,597 MB/s - MinLZ 155,804 MB/s. * HTML - zstd: 25,157 MB/s - MinLZ 82,292 MB/s. * URL List - zstd: 16,869 MB/s - MinLZ 45,521 MB/s. * GEO data - zstd: 11,837 MB/s - MinLZ 36,566 MB/s.

Of course zstd compresses to a smaller size - but for things like streams transfers or for fast readbacks from disk you probably want the fastest.

Overall, I would say that MinLZ is about 2x faster decompressing, and compresses about 2x the speed compressing at max speed. But of course with less compression.

I just released about 2 years of work on improving compression with a fixed encoding LZ77 style compressor.

Our goal was to improve compression by combining and tweaking the best aspects of LZ4 and Snappy.

The package provides Block (up to 8MB) and Stream Compression. Both compression and decompression have amd64 assembly that provides speeds of multiple GB - typical at memory throughput limits. But even the pure Go versions outperform the alternatives.

Full specification available.

In my experience, the hybrid approach is practically better due to cache locality anyway, so I don't think even native GF16 or GF32 would be of much help.

FWIW, I've ported rs-leopard to Go and found it very effective, except in the "recover one" scenario, where it is only at 50% speed of the "recover all" scenario, since it has to do several reconstructions to get one output. But even so, I am not too sure it would be much better with plain GF16, since you would still need to touch most shards to get one shard out.

Almost all (x86) CPUs sold have GFNI. That can pretty much saturate memory bandwidth on a single core or two. You can use SSSE3 pshufb for the rest which is about half the speed.

ARM has NEON and SVE/SVE 2. They also operate very fast.

So not sure what you are thinking of.

Looking at it quickly, it seems like you just moving entropy to the frequency table.

If the frequency table isn't included in the count, I could just make an "infinite" compressor by storing the frequency and cut one byte off the end. From the frequency table I could then deduce what the last byte should be.

> typical entropy encoders (Huffman, ANS, etc) would require 1 bit per symbol,

No. ANS (and range/arithmetic coding) allows for probabilities to be stored with fractional bits.

> If I could change one thing about S3's API I would like an option to read the metadata with the listings.

Agree. In MinIO (disclaimer: I work there) we added a "secret" parameter (metadata=true) to include metadata and tags in listings if the user has the appropriate permissions. Of course it being an extension it is not really something that you can reliably use. But rclone can of course always try it and use it if available :)

> You can create zero length files ending in /

Yeah. Though you could also consider "shared prefixes" in listings as directories by itself. That of course makes directories "stateless" and unable to exist if there are no objects in there - which has pros and cons.

> Or alternatively a way of setting the Last-Modified on an object when you upload it would do too.

Yes, that gives severe limitations to clients. However it does make the "server" time the reference. But we have to deal with the same limitation for client side replication/mirroring.

My personal biggest complaint is that there isn't a `HeadObjectVersions` that returns version information for a single object. `ListObjectVersions` is always going to be a "cluster-wide" operation, since you cannot know if the given prefix is actually a prefix or an object key. AWS recently added "GetObjectAttributes" - but it doesn't add version information, which would have fit in nicely there.