Comment by addedlovely

Nice one.

I've been doing the same bit wider scope, for the whole Crux list, pruned to apex domains, and looking for CMS signals - how's your throughput?

I'm not doing any headless browser stuff, or many requests, so hyper optimised for speed.

I do grab robots.txt - didn't really see much in llms.txt or humans.txt in the wild, does yours?