wrath

When websites get a ton of traffic the concern is that the algorithm that find fake data will not be accurate and start blocking paying customers. So it's a fine line between blocking paying customers and fake data. What these algorithms do instead of blocking is to throw captcha's so if the traffic is really human, the captcha can be solved. The bigger problem is that there's a good chance that humans who are thrown a captcha will leave to go buy somewhere else (either because they are lazy, the captcha is hard, etc...).

Solutions like CloudFare and Distill have sophisticated algorithm to balance out fake and real data but even they are not close to being perfect.

I can't provide details on any innovations we've done with sites like google, but in general if you want to crawl google you'll want to get "many, many" IP addresses. I've heard of people using services like 2captcha.com but the best way is to obfuscate who you are.

If you can hit Google 60 times per minute per IP before getting blocked and you need to crawl them 1000 times per minute, you need 17 IPs per hour. Randomize headers to look like real people coming from schools, office buildings, etc... Lots of work but possible.

Competitive intelligence.

Crawling thousands of website, mashing up the data to analyze competitiveness between them, and selling it back.

For example, cost of flights. Different websites provide different prices for the same flight. The technology crawls all the prices, combines the data, then resells it back to the websites. Everyone knows everyones prices, keeps competition high, lower prices for consumers.

I don't think I'm being hypocritical. I have no issues if people crawl my site, I even identify who they are and give them access to a private API. I do not generate any income from that website though. I provide a service because I love doing it and I cover all the costs. Bots do increase my cost so I choose to limit their activity. Crawl me, but do so using my rules.

One of the alternatives is charging for my service but bots are not my users problem, they are mine.

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will.

What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of pages, but there was a whole team dedicated to keeping it going). If you have money to spend, there's Distill Networks or Incapsula that have good solutions. They block PhantomJS and browsers that use Selenium to navigate websites, as well as rate limit the bots.

What I found really affective that some websites do is tarpit bots. That is, slowly increase the number of seconds it takes to return the http request. So after a certain amount of request to your site it takes 30+ seconds for the bot to get the HTML back. The downside is that your web servers need to accept many more incoming connections but the benefit is you'll throttle the bots to an acceptable level.

I currently run a website that gets crawled a lot, deadheat.ca. I've written a simple algorithm that tarpits bots. I also throw a captcha every now and then when I see an IP address hits too often over a span of a few minutes. The website is not super popular and, in my case, it's pretty simple to differentiate between a human or bot.

Hope this helps...

I think it's that's a very good rate of growth for a new service. The scaling challenges alone are not obvious and slowly taking up market share is the safer thing to do. If I was an investor seeing these growth rates, I'd be really happy as long as it keeps up and only plateaus when the goal is reached.

I think @nunobrito covered most of what I wanted to say.

Without knowing your product it's hard to give you the best advice, but here are things that worked for me in the past:

* Use real data that they can relate to. For example, if you're software does case management for non-profits. Find out who their customers are and create an "fake" example of how your software would work to manage a case for that company, from start to finish. This is partly for the 3 minute presentation, but more so for the barrage of questions you'll get asked afterwards once they like what they see. It's always easier to sell something when you find a solution to a pain your customer has right now, with one of his customers!

* Use only the terminology that they use and understand. Similar to #2, use the language that your customer speaks. Know your user and which language and acronyms they are using on a daily basis. I've seen so many times young startups use scientific terms or executive mumbo jumbo that the audience didn't understand.

* The 3 minutes is only the beginning. Whether you succeed or fail, it's not the end. Before, during, and after, identify the lead of the group that will be using your software and try to identify the decision maker. The person who signs the check if often different then the person evaluating your software. This is a sales cycle and they are both important for you to convince. Get a meeting with them, figure out what makes them tick, what their budget is, when will they be in a position to evaluate your software, etc..

Hope this helps a little, good luck!

I would politely insist that an email be sent out by the founder about your departure before the end of day. That way you get a chance of talking to your co-workers and explain the situation in your own words. To be clear, always be polite and never talk disparagingly about the company and co-founder, but the fact that you can't have a proper goodbye is a little strange. I've learned that it's always better to rip off the band-aid as quickly as possible while agreeing on an acceptable and honest message with the employee who is leaving. Better late than never I guess, so I would ask for this first thing this morning.

If it happens, send an email at the end of the day thanking everyone for the time you had with them, the knowledge you gain and the fun you had. Give your personal email and phone number in case there are any questions and/or someone wants to keep in touch with you.

As far as I know, shader.io sells individual proxies. Luminati and Nohodo are exit nodes priced by the amount of bandwidth that you use.

Sorry, my bad... it's shader.io

If these guys are serious about their business they are using proxies to obfuscate who they are. There are many services now that are claiming to have a ton of IP addresses Luminati, Shadio.io and Nohodo are just a few examples.

I love and miss RegexBuddy! If only they had a Mac/Linux version :( I've had thoughts about switching back to Windows while developing complex regular expressions.

I've built several businesses that either relied in-part to scrapping/indexing websites or solely relied on scrapping/index websites. I We never achieved the success of Google but we did get large enough to be noticed by some sites (Amazon for example). We did face legal issues but of a different kind. There were a few bugs early on that made us hit websites too much and we did receive a couple of cease and desist letters. We fixed our problem and explained the situation to the site owner and everything was resolved.

The only "fair use" type issue that we encountered was using logos from websites. E.g. Displaying the logos of the websites we indexes on our site. Once again, nothing serious came of it. I believe our marketing department removed the logos and put text instead.

Personally, I wouldn't worry about these issues until it becomes a problem. When it becomes a problem it means you're on to something and you're disruptive enough to get some attention. It's a good problem to have IMO.