TonyTrapp
Joined 6 years ago
Karma
3,945
Submissions
822
Member since
Jun 25
Joined 6 years ago
- Respects robots.txt
- Detects if something is likely a git forge and as a result
- Clone the git repository instead of trying to crawl the forge
None of this needs to be happening, and yet it does at an unprecedented scale.