Comment by BorisMelnik

no you are def right the agents are def capable and that is the problem, so capable we are all doing this, we are all making our own scrapers, as are big corps/small corps and it is adding up so now we are seeing managed challenges everywhere and webmasters/sysadmins going to great lengths to block any bots.