Comment by xena

You should also support Web Bot Auth and make it required to have a customer-specific abuse contact in the user-agent header so that people can send invoices to those customers for when your customers cause downtime due to excessive scraping.