Comment by tompec

It does respect robots.txt when crawling. I'll add more details about this in the docs.