Comment by wizofaus

Doesn't mention the hardest part I found when developing a crawler - dealing with pages whose content is mostly dynamic and generated client side (SPA's). Even using V8 it's hard to do reliably and performantly at scale.