Hey HN,
I just built a small web app that helps you generate /llms.txt files for your website:
https://llmstxt-cyuh.vercel.app
GitHub: https://github.com/mandarwagh9/llmstxt
What is /llms.txt?
Think of it like robots.txt, but for large language models. It’s a new emerging standard that gives LLMs a clean, Markdown-based map of your most important content (docs, guides, APIs). Instead of crawling bloated HTML, they get straight to the point.
The idea was popularized by Jeremy Howard and is gaining adoption across devtool companies like LangChain and Mintlify.
Why I built this
I wanted a dead-simple way to auto-generate /llms.txt from any docs site or URL. This tool scrapes your site, extracts key Markdown links, and gives you a copy-paste-ready llms.txt.
No setup, no install, just paste your URL and download.
Stack
Frontend: Vercel + vanilla JS
Backend: Uses Gemini API (but can swap in any LLM)
Built with a focus on minimal UI + performance
Would love feedback, feature ideas, or suggestions for edge cases to cover (like private docs, multi-language support, etc.)
Thanks!
I just built a small web app that helps you generate /llms.txt files for your website: https://llmstxt-cyuh.vercel.app GitHub: https://github.com/mandarwagh9/llmstxt
What is /llms.txt? Think of it like robots.txt, but for large language models. It’s a new emerging standard that gives LLMs a clean, Markdown-based map of your most important content (docs, guides, APIs). Instead of crawling bloated HTML, they get straight to the point.
The idea was popularized by Jeremy Howard and is gaining adoption across devtool companies like LangChain and Mintlify.
Why I built this I wanted a dead-simple way to auto-generate /llms.txt from any docs site or URL. This tool scrapes your site, extracts key Markdown links, and gives you a copy-paste-ready llms.txt.
No setup, no install, just paste your URL and download.
Stack
Frontend: Vercel + vanilla JS
Backend: Uses Gemini API (but can swap in any LLM)
Built with a focus on minimal UI + performance
Would love feedback, feature ideas, or suggestions for edge cases to cover (like private docs, multi-language support, etc.)
Thanks!