Robots.txt checker and tester
Test any URL against a site's robots.txt for Googlebot, Bingbot and AI crawlers like GPTBot, ClaudeBot and PerplexityBot.
Free to use, no signup needed.
Test robots.txt rules before they cost you traffic
One wrong line in robots.txt, such as Disallow: / left over from a staging site, can stop search engines crawling your whole website. This tester fetches the live file, applies the rules for each crawler to the exact URL you enter, and tells you whether Googlebot, Bingbot, Applebot, DuckDuckBot and the main AI crawlers may fetch it.
It follows the robots exclusion standard (RFC 9309): a crawler uses its own user-agent group if one exists, otherwise the * group, and the longest matching rule wins, with Allow winning ties. Wildcards (*) and end anchors ($) are supported.
Want to know how AI search engines see the rest of your site? Run the AI visibility checker, and create an llms.txt file to point them at your best pages.
Frequently asked questions
What does robots.txt do?
robots.txt is a plain-text file at the root of your site that tells crawlers which paths they may fetch. It controls crawling, not indexing: a blocked page can still appear in Google if other sites link to it. Use a noindex tag to keep a page out of results.
How do I test a specific URL?
Enter the full URL, for example yoursite.com/blog/post. We fetch robots.txt from that site and evaluate the most specific Allow or Disallow rule for each crawler, using the same longest-match logic Google uses.
Should I block AI crawlers like GPTBot?
It depends on your goals. Blocking training crawlers (GPTBot, Google-Extended, CCBot) keeps your content out of model training. Blocking search crawlers (OAI-SearchBot, PerplexityBot) also stops those assistants citing you. The table shows each one separately.
Does Google support crawl-delay?
No. Googlebot ignores Crawl-delay. Bing and Yandex respect it. Adjust Google's crawl rate in Search Console if you need to.