Comment by jshen

The hard part is understanding which parts are the content versus navigation or promotions of other content.

I’ve written a couple search engines. Have you tried making one with beautiful soup?