What kind of websites? You mean like social media sites that are obfuscated to prevent scraping? I suppose, it would have to be quite reliable.
I don't know how relevant this is, but I was thinking that you could probably use some sort of AI to enhance OCR and convert written documents into some sort of semantic form like HTML or Latex. That would allow you to use books to scrape information, and written books still have a lot of untapped knowledge.
It seems like the demand for web scraping and such is to create datasets for ML training. And now you are using AI for scraping. So it is sort of a self-improving cycle
I don't know how relevant this is, but I was thinking that you could probably use some sort of AI to enhance OCR and convert written documents into some sort of semantic form like HTML or Latex. That would allow you to use books to scrape information, and written books still have a lot of untapped knowledge.
It seems like the demand for web scraping and such is to create datasets for ML training. And now you are using AI for scraping. So it is sort of a self-improving cycle