Comment

Comment by marginalia_nu

As a search engine operator, I can confirm webmasters put a lot of weird crap in robots.txt. ASCII art, poems, emoji, love letters, sudoku puzzles, philosophical questions, whatever. It's cute, but odds are at least some parser will choke on them and depending on how the crawler is coded, either ignore your robots.txt; or skip crawling your site. It's not great either way as the format is more of a set of conventions than a proper well defined language all specified up in BNF.

Webmasters also put highly specific URLs they don't want the world to see in robots.txt. Also not a great idea.

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton