I think the problem we have is an ill-defined concept of "human consumption". I understand the desire (instinct?) to ban crawlers, but I'd argue that bots serving human needs are different. Consider: bots that fetch the content for me so as to present it in a different form. Bots that need to fetch documentation, because they are writing software for me. Bots that read or explain the content to me because I am disabled. There are many use cases where "bots" are there specifically to fulfill human needs, and yet we ban them.
Worse, many of us outsource the control over who can access their website to a third party, in this case a corporation that is an increasingly menacing monopoly on the internet — which I think is a problem, too.
Worse, many of us outsource the control over who can access their website to a third party, in this case a corporation that is an increasingly menacing monopoly on the internet — which I think is a problem, too.