Comment by TonyTrapp

So? You can design a deterministic crawler for AI that

- Respects robots.txt

- Detects if something is likely a git forge and as a result

- Clone the git repository instead of trying to crawl the forge

None of this needs to be happening, and yet it does at an unprecedented scale.