Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea.
I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.