Comment

Comment by hiccuphippo

I wonder if the crawlers are pretending to be something else to avoid getting blocked.

I see Bun (which was bought by Anthropic) has all its documentation in llms.txt[0]. They should know if Claude uses it or wouldn't waste the effort in building this.

[0] https://bun.sh/llms.txt

parent hiccuphippo 8 months ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton