Lasang
Joined 2 years ago
Karma
16
Submissions
17
Member since
Oct 14
Joined 2 years ago
If an agent runs unattended for hours, small errors compound quickly. Even simple misunderstandings about file structure or instructions can derail the whole process.
If more sites provided explicit machine-readable entry points for crawlers, indexing could become a lot less wasteful. Right now crawlers spend a lot of effort rediscovering the same structure over and over.
It also raises interesting questions about whether sites will eventually provide different views for humans vs. automated agents in a more formalized way.