robn_fastmail

I don't know about bounties, because I'm not personally in favour of vigilantism, but I do take your point.

Honestly, I think the ease in which people can be anonymous is major problem here. Anyone with an internet connection can buy botnet time with Bitcoin and accept a ransom in the same way. It makes it incredibly difficult to follow the path back to the attacker.

At this point pretty much the only thing you can do is collect data and share it with CERT and other relevant law enforcement. I don't have a good sense of how effective they can be, but it makes sense that the more data they have, the better chance they have at identifying specific botnets and follow the path back to the owners.

Generally, we talk about what we're doing because we're all excited about what we do. It's always been that way for us, and our customers really appreciate the honesty and transparency.

On this particular one, we really did learn a lot and we were keen to share some of that. It's really difficult to run an internet service and we feel we have a duty to try and make this easier, or at less better-understood, where we can.

Our business can't exist with a large diverse network that anyone can get involved on, and we couldn't have got to this point without the knowledge of others, whether that's embedded in the open-source software we run or in the blog posts and emails of other people that figured out hard stuff. It wouldn't be right for us to take and not give something back in return.

There's also an element of defiance in this post too. We got punched in the face. We're not going to respond by hiding in a corner. We're going to say "you know what, fuck you" and we're going to help (and have helped) others to do the same in whatever way we can.

Quite. As my dear colleague said this morning, "hubris" is the word of the day. Still, I'm not sorry we posted it.

Datacentre confirmed a DDoS attack, which took a little while to mitigate. We're looking good now, but we're continuing to monitor.

Indeed. As I note in the post, this wasn't something we _needed_ to do at that time, we were just curious.

Replacing a function with a faster implementation is trivial; its just another deployment and we do several of those each week. Changing data format adds operational complexity - two codepaths need to be run and maintained and we increase system load and lower response times for the duration. (The data format is versioned, so that's easy - no architectural problem there).

Its totally worth doing if you need to, and we're not afraid of that, but you don't do it on a whim - it takes proper planning and testing and needs multiple people involved.

The results in the tests are all with optimisations (-O3 -march=sandybridge -mtune=intel), as mentioned in the post.

The exception is the Debian packaged version of zlib, because we don't control that. That's the reason I include stock zlib in the tests - if it had been wildly different from the system zlib, I would have looked into recompiling the Debian package with more optimisations. There were no major differences and indeed, inspecting the package further shows that it is compiled with optimisations.

On the Cyrus side, we used to link to the Debian zlib, so we get those optimisations. For the new code bundled in Cyrus itself, we have to enable optimisations ourselves.

I think that mostly, your benchmarks have to match your workloads. Most of the CRC32 benchmarks I've seen are looking at larger buffers. The xxhash function mentioned elsewhere in this thread was claimed to be "an order of magnitude" faster, but again, large buffers - the gain over CRC32 on the same tests were rather more modest (though not at all insignificant).

In this case, I think (but am curious, will investigate further at some point) our Cyrus servers are doing enough checksumming work to keep any tables hot in the cache. So the tests are hopefully a useful indicator of where improvements can be made.

My quick tests here (using the same methods outlined in the post) suggests its around 30% faster on 64-byte buffers. If we're ever shopping for a new hash function entirely, I'll make sure its considered. Right now its not worth it because as noted in the post, we're not actually under any particular stress and changing hash functions is a big job.

They do work, but only if you upload them via CardDAV - we don't have any UI for them in the web client yet.

Not a chance :)

In what way do you consider the privacy policy cavalier? What would you prefer it said?

I don't believe we made any claims that Australia is a "privacy Eden" or that we're "paragon of privacy". Indeed, we frequently say we're _not_ a privacy service, just an email service that cares about privacy among other things.

More specifics on both these points would help us discuss them properly.