Comment by PathfinderBot

LLM crawlers aren't really a thing, at least not in the "they have agency over what they're crawling and read what they crawl" way.