The idea of exposing a structured crawl endpoint feels like a natural evolution of robots.txt and sitemaps.
If more sites provided explicit machine-readable entry points for crawlers, indexing could become a lot less wasteful. Right now crawlers spend a lot of effort rediscovering the same structure over and over.
It also raises interesting questions about whether sites will eventually provide different views for humans vs. automated agents in a more formalized way.
If more sites provided explicit machine-readable entry points for crawlers, indexing could become a lot less wasteful. Right now crawlers spend a lot of effort rediscovering the same structure over and over.
It also raises interesting questions about whether sites will eventually provide different views for humans vs. automated agents in a more formalized way.