Comment by mfost

I thought the point (which the article misses) is that a token gives you an identity, and an identity can be tracked and rate limited.

So a crawlers that goes very ethically and does very little strain on the server should indeed be able to crawl for a whole week on a cheap compute, one that hammers the server hard will not.