For anyone managing content and worried about AI crawlers, I recently put a guide together with notes about the most common AI crawlers, and how to manage them! www.aibrandrank.com/guides/robots-txt-for-ai
•
In response to @michaelcrowcroft
I saw this on Hacker News today which is potentially related (llms.txt): news.ycombinator.com/item?id=41439983
•
I saw that! It's an interesting idea, although I think ultimately a bit of a redundant idea for most websites.