Comment by nerdo

You can't legally make a crawler to compete with Google anymore, it's explicitly stated in a lot of robots.txt's.