Ask HN: Have there been any innovations in crawling or data extraction?
I'm currently working on a project which involves crawling data from a large collection of public sites. I need to figure out how to crawl these sites and parse the data that I need.
In the past I've used tools like selenium and beautiful soup to crawl and extract data. A lot of the tedious work is in manually figuring out how to crawl sites and extract data from pages, which is particularly annoying when you have numerous different sites to manage.
My knowledge is a bit outdated. Are there any tools today that make this kind of work easier? Anything that can automatically figure out how to crawl a site or automatically extract data from a page?
In the past I've used tools like selenium and beautiful soup to crawl and extract data. A lot of the tedious work is in manually figuring out how to crawl sites and extract data from pages, which is particularly annoying when you have numerous different sites to manage.
My knowledge is a bit outdated. Are there any tools today that make this kind of work easier? Anything that can automatically figure out how to crawl a site or automatically extract data from a page?