llms.txt for AI Crawlers: Why robots.txt Is Not Enough

Originally published at chudi.dev llms.txt is a site-level policy file that tells AI engines how they can use and cite your content. It complements robots.txt by focusing on usage and attribution rather than crawl access. If you want AI systems to c...