Comment by mtmail

Do the scripts respect the https://twitter.com/robots.txt ? It contains

    User-agent: *
    Disallow: /