Comment by ushakov

> We don't spin up a full browser instance for every request (which is slow and expensive)

there's really no excuse for not spinning up a browser every request. a Firecracker VM boots ~50ms nowadays

> We respect robots.txt rules.

you might, but most companies in the market for your service don't want this