Loading...
Loading...
See which AI bots your robots.txt lets in, and whether you have llms.txt
AI companies use separate crawlers for different jobs: some collect training data, some power AI search results, and some fetch a page when a user asks about it. Your robots.txt decides which of them can read your site, and a single wildcard rule can block or allow all of them without you noticing.
Enter a domain to read its robots.txt and see, for 14 well-known AI crawlers, whether each one is allowed, partly allowed or blocked, and whether that comes from a rule naming the bot or from the wildcard group. The checker also looks for an /llms.txt file.
To see whether AI assistants actually cite your site, try the AI Visibility Tracker.
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot and Bytespider gather data that can be used to train models.
OAI-SearchBot, Claude-SearchBot, PerplexityBot and the -User agents fetch pages to answer questions, often with a link back to you.
'via *' means no group names the crawler, so it follows your User-agent: * rules. Add a named group to treat it differently.
Without a robots.txt every crawler is allowed by default. That's fine if it's what you want, but it's worth deciding on purpose.
It depends on your goals. Blocking training crawlers such as GPTBot, CCBot or Google-Extended keeps your content out of future model training. Blocking search and user-triggered agents such as OAI-SearchBot, ChatGPT-User or PerplexityBot can also stop those assistants from citing and linking to your pages. Many publishers block training while allowing search.
According to Google, Google-Extended controls whether content is used for Gemini models and grounding. It does not affect Googlebot crawling or inclusion in Google Search, and AdSense uses its own crawler (Mediapartners-Google).
The crawler's rules allow some paths and disallow others, for example Disallow: /members/. It can still read most of the site, but not everything.
llms.txt is a proposed convention, not an official standard: a Markdown file at /llms.txt that gives language models a short summary of a site and links to its key pages. Support among AI companies is limited and unconfirmed, so treat it as optional.
robots.txt is a voluntary standard. The major companies listed here say their crawlers honour it, but it isn't enforced technically. To block a crawler outright you need server or firewall rules, such as Cloudflare's bot controls.