Comment by latentdeepspace

Can someone provide a bit of background how the crawling part works?