At a previous SaaS startup I worked for, we stored a bunch of clients' documents somewhere on e.g. example.com/documents - which included confidential items like checks and contracts. Customers would typically opt-out of requiring authentication for a lot of these documents so they could share them with others.
I noticed that when you searched for the company on Bing, you would actually see a bunch of these documents, despite nothing linking to it!
Of course I updated the robots.txt and yelled at leadership for the gaping security hole, but I was very surprised to see that Microsoft would send every link you crawled back to Bing to index. Distributed web crawling!
I noticed that when you searched for the company on Bing, you would actually see a bunch of these documents, despite nothing linking to it!
Of course I updated the robots.txt and yelled at leadership for the gaping security hole, but I was very surprised to see that Microsoft would send every link you crawled back to Bing to index. Distributed web crawling!