Free tool
Can ChatGPT, Claude and Perplexity read your site?
Enter your address. We test 14 AI and search crawlers against your robots.txt, your server and your page tags, and show which assistants can actually get through.
Three places a crawler gets turned away
Most checkers only read robots.txt. That misses the blocks that are set outside the website, which are the ones owners never see.
robots.txt rules
We read your robots.txt the way the crawlers do, following the published standard: the group that names the bot wins, and the longest matching rule decides. A single line can shut out ChatGPT while leaving Google alone.
The server itself
robots.txt is a request, not a wall. We then ask your homepage for a page while identifying as each crawler, and compare the answer with a normal browser. A firewall rule, a Cloudflare setting or a host limit shows up here even when robots.txt says yes.
Page tags and llms.txt
A robots meta tag or X-Robots-Tag header can tell search engines not to index a page they are allowed to fetch. We also note whether you publish an llms.txt, which some assistants read and Google ignores.
Questions
Split the question in two. Crawlers that fetch pages to answer a question (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot and their user agents) decide whether an assistant can mention and link you, so blocking them costs visibility. Crawlers that only gather training data (GPTBot, ClaudeBot, Google-Extended, CCBot) are a genuine choice: blocking them does not remove you from AI answers.
Because something in front of the website refused the request. robots.txt only asks crawlers to stay away; a firewall rule, a Cloudflare bot setting or a hosting provider's limit can reject them outright, and those are set outside the site, often by default. The server test column shows the HTTP status your site returned to that crawler.
Close, not identical. We send a request that identifies itself as each crawler. Some sites verify a crawler by its network address, so a site that admits the real bot may still refuse our test. Treat a server block as a strong hint and confirm it in your firewall or hosting logs. If your homepage refuses even an ordinary browser request, we mark the server test as not possible rather than guess.
No. It is optional. Google Search ignores it. Some assistants read it as a plain summary of what the site is, so it can help them describe you accurately, but it does not unlock anything a blocked crawler cannot already see.
More free tools
Redirect Checker
Every hop from an address to its final page, plus a check that http, https, www and non-www all end in one place.
Run the checkRobots.txt Generator
A robots.txt for Shopify, WordPress or WooCommerce, with one-click rules for AI crawlers and a warning before a wrong line hides your site.
Run the checkSitemap Checker
Finds your sitemap, counts the addresses and opens a sample of them to catch dead, redirected and noindexed pages.
Run the checkBlocked, and not sure where it is coming from?
Send us the address. We find which layer is turning the crawlers away and fix it, without opening the site to the scrapers you actually want kept out.
No pitch deck, no obligation. Bring your URL and we will look at it together.