>> interesting. workers.dev domains can be a liability sometimes -- if you've mapped the worker to a real zone, then you probably don't want the workers.dev zone anymore.
maybe that is a good idea, prompt a user to disable the worker.dev domain. however, having the worker.dev domain open for me is more of a backup way to access the worker as the mapped domain is more for a hobby project.
> Interesting. workers.dev domains can be a liability sometimes
what about allowing the specific worker.dev to serve a robots.txt on its own that can be used to disable the AI bots?
maybe that is a good idea, prompt a user to disable the worker.dev domain. however, having the worker.dev domain open for me is more of a backup way to access the worker as the mapped domain is more for a hobby project.
> Interesting. workers.dev domains can be a liability sometimes
what about allowing the specific worker.dev to serve a robots.txt on its own that can be used to disable the AI bots?
i.e https://XXXXXXXXXXX.username.workers.dev/robots.txt should be allowed to be configure on a worker level. not sure how that affects the design of the infra, but it would be a good idea