Converting websites to markdown comes with 3 distinct problems:
1. Throughly scraping the content of page (high recall)
2. Dropping all the ads/auxilliary content (high precision)
3. And getting the correct layout/section types (formatting)
For #2 and #3 - Trafilatura, Newspaper4k and python-readability based solutions work best out of the box. For #1, any scraping service + selenium is going to do a great job.
Could you elaborate on what your tool does different or better? The area has been stagnant for a while. So curious to hear your learnings.
1. Throughly scraping the content of page (high recall)
2. Dropping all the ads/auxilliary content (high precision)
3. And getting the correct layout/section types (formatting)
For #2 and #3 - Trafilatura, Newspaper4k and python-readability based solutions work best out of the box. For #1, any scraping service + selenium is going to do a great job.
Could you elaborate on what your tool does different or better? The area has been stagnant for a while. So curious to hear your learnings.