Comment

Comment by firloop

Interesting. So, it seems like you aren't respecting robots.txt. I picked Old Navy, as it was on your supported stores page [0], and went to their robots.txt [1]

    User-agent: *
    Disallow: /buy/
    Disallow: /checkout/

So, do you have permission to violate robots.txt, as I'm sure there is some automated interaction with checkout/purchasing pages? Or I am I missing something about how TwoTap works? Scraping is one thing, but accessing when the management of the website prohibits it seems like a big no no.

[0] : https://twotap.com/supported-stores/

[1]: http://oldnavy.gap.com/robots.txt

parent firloop 12 years ago

Comment by firloop

Discussion