Free tool
Robots.txt generator
Pick your platform, add your sitemap, and choose which AI crawlers to block. We write the file and warn you about lines that could hide your site.
What is your site built on?
On Shopify you edit this in your theme as robots.txt.liquid. Paste only the rules you want to add to Shopify's defaults.
Your robots.txt
User-agent: * Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /search Disallow: /*?*sort_by=* Disallow: /*?*filter.*
- No sitemap address. Adding one helps search engines find all your pages.
Upload it to the top level of your site so it loads at yourstore.in/robots.txt. Then run the AI crawler checker to see what each bot is allowed to read.
How we can help
One wrong line can hide your whole site
A robots.txt file is short, and a single mistake in it can keep search engines and AI assistants away from your best pages. We write it for your platform and check it against what is actually being crawled.
A robots.txt written and tested for your store, with AI crawler rules that match your goals.
From ₹6,000See the service AI Search OptimizationMake your site readable and quotable by ChatGPT, Claude, Perplexity and Google's AI answers.
From ₹7,999/moSee the service Technical SEOCrawl, index and redirect problems found and fixed across the whole site.
From ₹7,999/moSee the serviceNot sure which you need? Book a free 30-minute call and bring this result, or send us a message. No obligation, no pitch deck.
Questions
A small text file at the top of your site that tells crawlers which parts they may visit. Search engines and AI crawlers read it before they read anything else.
At the root of your domain, so it loads at yourstore.in/robots.txt. On Shopify you edit it in your theme as robots.txt.liquid. On WordPress and WooCommerce you can upload the file, or use your SEO plugin's robots.txt editor.
Blocking training crawlers such as GPTBot, ClaudeBot or Google-Extended does not change how you rank in Google search. Blocking the AI search and assistant crawlers can keep you out of AI answers, so block those only if that is what you want.
No. It asks crawlers not to visit a page. A blocked page can still appear in results if other sites link to it. To keep a page out of search, use a noindex tag, and leave the page crawlable so the tag can be read.
No. It is a request, and well-behaved crawlers follow it. A bad bot can ignore it, and the file itself is public, so never list a private address in it.
No. The file is built inside your browser. Nothing you enter is sent to us.
Guides
More free tools
AI Crawler Checker
Can ChatGPT, Claude, Perplexity and Google read your site? 14 crawlers tested in ten seconds.
Run the checkRedirect Checker
Every hop from an address to its final page, plus a check that http, https, www and non-www all end in one place.
Run the checkSitemap Checker
Finds your sitemap, counts the addresses and opens a sample of them to catch dead, redirected and noindexed pages.
Run the checkNot sure what your robots.txt is blocking?
Send us your site. We check what search engines and AI crawlers can actually read, before there is any quote.
No pitch deck, no obligation. Bring your URL and we will look at it together.