Comment by Grimm1

I'd say most crawlers are looking to provide enriched value for content at their end use. Google is just an aggregator (the biggest by far) but other aggregators are looking to provide similar value.