laCour

I'm with Hund. It was only monitoring the cached, unauthenticated news page, which was up throughout the downtime. However, it now monitors the authenticated news landing as well.

Mistake on our part (Hund) for not monitoring authentication. This page is unofficial and was made by a co-founder several years ago.

This was monitoring the unauthenticated news page, which is why it didn't catch it. It now monitors authentication as well. It is not official, and was made by a co-founder years ago.

I'm with Hund. Our hn.hund.io page did not catch this because it was requesting the cached, unauthenticated page. It now monitors authentication as well.

They only just clarified that monitoring was in place and they were reporting as healthy. See the comments above.

Right, but they didn't recover speedily. To have the cluster in such a state for so long sounds like poor monitoring to me because this can knowingly interfere with an election later.

"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons."

How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.

If I'm not mistaken, Rdio's engineering team helped build Pandora's on-demand service. I've switched to it from Spotify after using Rdio in the past. Its recommendations feel similar to Rdio's, but the interface is a bit lacking. There aren't general recommendations. Instead, you must build a playlist and then you can add a set of recommended songs to the playlist. Of course, there are also radio stations.

13:14 UTC through 15:54 UTC here.