A really cool story. But I have to say, everyone will notice that rewriting a file transfer tool in Rust is a poor use of the engineering time without having first understood the cause of the slowness. It's almost like a blind cult-like trust in Rust.
> So just like with everything else, we decided to write our own, in Rust. After days of digging through Tokio documentation and networking theory blog posts to understand how to move bytes as fast as possible between the filesystem and an HTTP endpoint, we had a pretty basic application that could chunk a byte stream, send it to an object storage service as separate files, download those files as they are being created in real time, re-assemble and pipe them into a ZFS snapshot.
I mean this sounds like a fun engineering project and I suspect I would enjoy writing it very much. While this might be bring me joy personally, as an organization this is still a failure.
> So just like with everything else, we decided to write our own, in Rust. After days of digging through Tokio documentation and networking theory blog posts to understand how to move bytes as fast as possible between the filesystem and an HTTP endpoint, we had a pretty basic application that could chunk a byte stream, send it to an object storage service as separate files, download those files as they are being created in real time, re-assemble and pipe them into a ZFS snapshot.
I mean this sounds like a fun engineering project and I suspect I would enjoy writing it very much. While this might be bring me joy personally, as an organization this is still a failure.