Comment by xhrpost

Is this what is causing at least part of the massive bot traffic hosts are seeing to their sites? Many direct-to-inference users are just retrieving the site locally every time they need it, let alone sharing site content across users. The big providers like Claude and Google implement their own caching so that when an agent is doing research, if 100k different prompts need the same site content, they don't hit that site 100k times.