Comment by pmohun

It will be interesting to see the next version of robots.txt that begins to appear in public repos to prevent this type of data scraping.