Comment by dscpls

Why a service and not a library?

It looks like a great way for you to discover URLs but like a terribly slow way for people to avoid implementing robots.txt rules.