Recent submissions

Comment · 4 years ago
Unfortunately not - the links were indexed and shown in Bing search results
Comment · 4 years ago
I do have a robots.txt to block this directory. But Bing only listens to that for what to crawl, not what to index.
Comment · 4 years ago
I’m fairly certain they are. My links ended up indexed in Bing search results. The only place they were ever rendered was in private emails to users. Bing should not be indexing that.
Comment · 4 years ago
Exactly that.
Comment · 4 years ago
This is very useful information. You’d really hope that private emails would be excluded by default…
Comment · 4 years ago
The only purpose of this link was to verify that the email address is valid. Once it’s verified, you can login.
Comment · 4 years ago
Many people (most?) prefer to signup to services by email address. To do so, those email addresses must be verified. How would you verify it without sending them an email link?
Comment · 4 years ago
You’re right that it was a bit of an oversight on my behalf, as the links were only generated after a verified human user action (signup) I had assumed the 1 time links to their email would be safe. But regardless of the link action, it shouldn’t be passing that data to Bingbot to crawl and (possibly) index in search engine results. Private email data should not be shared with search engine crawlers IMO.
Comment · 4 years ago
I would be pretty mortified if browsers were using user browser sessions to scan content and pass it to bingbot…? What about if you’re browsing something local? Or your bank account?
Comment · 4 years ago
Comment · 4 years ago
Exactly - seems the safest way is to explicitly block known bots by user agent from even reaching pages you don't want indexed.
Comment · 4 years ago
You may be seeing a stale version, try this: https://shoprocket.io/robots.txt?bypass=1

(I made a lot of changes today when testing all, including "visit as Bingbot" from their webmaster tools with and without the URL blocked by robots.txt)

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton