mkjones

I had a similar but less-awful experience (fortunately as a renter rather than owner). Living in a similar slab house, I was surprised when my shower became slow to drain and draino had no effect. I called the plumber, he came by, and eventually told me he'd fixed the problems. Two days later, the drain backs up again.

Second plumber comes and is somewhat perplexed - he pulls up roots and suspects they've grown into the line. But it's drilled out and should work now. Two days later, the drain backs up again.

Third plumber comes out and goes in from the roof vent. After about an hour of rooting around, his snake pops up through the drain and starts whackkity-whacking around the shower. He's very surprised and a bit sheepish, and suggests there's a bigger problem and we should rip out the shower to have a look at what's under it. Doesn't pretend that he's fixed it. I'm using a different shower at this point.

Fourth plumber arrives to give a second (err, fourth) opinion and is similarly perplexed, saying we likely need to rip the shower up to fix things.

Fifth plumber agrees, and says he knows a guy who can do it.

Sixth plumber is actually not just a plumber, and arrives with a jackhammer, two buddies and a fancy proble that lets him follow where all the pipes go under the house (both with a camera in the pipe and a machine that follows where it goes physically). Scopes things out for a bit, and then goes to town on the other bathroom, eventually jackahammering and digging a roughly 5 foot cube out from under the bathroom.

It turns out when they redid things in the house 15 years ago, someone failed to actually connect the drain from the shower to the sewer, so it had just been draining into the ground. Whoops!

While they were down there, they ran a new sewer pipe out to the municipal line as well. This part was pretty cool. Rather than digging up the whole yard, they just ran a 1/2" steel cable through the old masonry pipe, hooked it to a solid steel cone, and pulled the cone through with a hydraulic winch, trailing some pvc behind it. As it went, it busted apart the old pipe and put a new one its place. All in all a pain in the ass, but neat to see how they approached things.

Aside: I also have and love Amica. They've covered a couple claims for me and have been very easy to work with.

OK, I can think of a few ways they might do this (DNS tricks and per-user IPv6 addresses, <src ip, src port, dst ip, dst port> => user mapping). These all seem significantly more complex than HTTP header injection though.

How would ISPs tag requests going over HTTPS as being from a particular subscriber?

Yep, "URLs shared primarily by not-sharey people" sounds similar to a lot of classifiers we have.

Yeah, we've played with that idea a bit. It doesn't help the sparseness problem (actually makes it worse), and if we took action as a direct result, it would give people the power to DoS content they disagree with.

For sites operating at a smaller scale, this could be a good way to surface content for manual review though.

So I can't speak for twitter, but I work on anti-spam at Facebook, and imagine the problems we face are relatively similar. It's worth noting that there's a constant barrage of people trying to send varying degrees of spam. It's not like there's An Attack all of a Sudden - just occasionally people close to the HN social network happen to be targeted by something and it's magnified by the media / hive mind local to us.

> shouldn't Twitter be able to pick these messages up automatically fairly fast

Theoretically, sure. As a human looking at an attack, it's usually pretty easy to pick out "obvious" attributes that they should have been able to catch. But when you're operating at a scale like us or Twitter, even stuff that looks like it's obviously-indicative-of-badness often has false-positives (posts flagged as spam that are not). The long tail of weird stuff that a billion users do can be pretty crazy.

At the same time, the "obvious" attributes of an attack are often very cheap for an attacker to change. Instead, we try to go after more expensive resources (domains, source IPs, etc).

> after (I assume) hundreds if not thousands of users have flagged them

Sadly, looking at flags of content is not a silver bullet. The signal is very sparse (a given spam post is rarely flagged), and nonspam posts are frequently flagged (religious and political speech are great examples - and they are the worst kind of false positive if you delete them as spam). These problems can be somewhat mitigated if you aggregate flags over a dimension that's expensive for the attacker (domain-posted, IP that posted the content, text shingles), but even then the recall isn't necessarily great and you could still catch e.g. controversial political domains.

> the spammers can't have unlimited IPs

True, though you can rent space on a botnet that has many, geographically-diverse, real-user IPs. Also, I imagine a significant chunk of posts to Twitter come from apps, many of which each use a single IP to post tons of content.

> Is there a reason the same techniques used in E-Mail aren't applicable to Twitter?

There's definitely some overlap. I'm not an expert at email anti-spam, but in general it's a relatively different problem. "Traditional" email spam is sent from some random email address on / via a compromised machine or open relay, and seems to be a relatively-well-solved. But it sounds like this twitter attack was caused by compromised accounts. At least anecdotally, it seems that email vendors are also not great at detecting this kind of attack. For example, my gmail account (with arguably the best spam protection in the industry?) gets a message every few weeks from some compromised friend's account. (i.e. someone had their email password stolen and the attacker is using it to "legitimately" send mail after authenticating to that email service with the correct password).

I'm surprised more aren't mostly-white for the first 3/4, and then they split at the end. It seems like all the action in basketball always happens in the last 2 "minutes."

We (I'm another eng at FB) definitely hire interns who are not juniors in college. You can be younger or older (or you could be a junior - I did my internship here after junior year).

As I understand it you just need to be going back to school at the end of your internship.

Hmm, wanna report at facebook.com/whitehat with more details? Please include repro instructions :).

I think there's a spectrum between letting whitehats do anything (including violating privacy, hurting real user accounts, etc) vs. suing everyone who changes a GET param somewhere. Having a whitehat program with (IMO reasonable) guidelines around not impacting unsuspecting real users seems to me like a good balance and is fairly close to the first part of the spectrum.

Obviously I don't love the end outcome, and this would have gone better for all parties if he had used a test account and included some kind of repro instructions (like that video) in the initial report.