I think llm.txt was invented as a way for AI to read what the page says in an optimized way. AI doesn't need layout, animation, media and other fluff. The main problem with llm.txt is that it can be considered as a separate page (indexed) and needs separate maintenance (likely to go out of sync). That is why it likely won't be a standard.
I think, the standard would be an automatic support of markdown content negotiation [1]. The only issue is that page content is written for humans to consume and AI doesn't need most of it.
I think, the standard would be an automatic support of markdown content negotiation [1]. The only issue is that page content is written for humans to consume and AI doesn't need most of it.
1. https://blog.cloudflare.com/markdown-for-agents/