Free tool
A correct robots.txt in a minute, tested before you upload it.
Start from a preset, add your own rules and sitemap, and test any URL against the file with the same matching rules Google uses.
What you get
- WordPress, shop and staging presets
- Sitemap line and AI-crawler block
- Built-in URL tester
Only want to manage AI bots? Use the AI robots.txt builder.
Extra rules for all crawlers
User-agent: * Disallow:
Test a URL against this file
Allowedno rule matches
Upload the file to your site's root so it opens at example.com/robots.txt. robots.txt asks crawlers politely; it does not hide pages. Use a password or noindex for anything private.
robots.txt, briefly
It tells crawlers where not to go. It doesn't hide anything.
robots.txt sits at your site's root and asks crawlers to skip certain paths. A single wrong line, like Disallow: / left over from a staging site, can remove a whole site from Google.
- Where does robots.txt go?
- In the root of your domain, so it opens at example.com/robots.txt. Each subdomain needs its own.
- Does robots.txt stop a page appearing in Google?
- Not reliably. A blocked page can still be listed if other sites link to it. To keep a page out of search, allow crawling and add a noindex tag, or put it behind a password.
- Which rule wins when Allow and Disallow both match?
- Google uses the longest matching path. If they're the same length, Allow wins. The tester on this page applies the same rule.
- Should I block AI crawlers?
- It's a business choice. Blocking training crawlers like GPTBot keeps your content out of future model training, but blocking AI search bots can also keep you out of AI answers.
- Got a sitemap?
- Make one with the XML sitemap generator and add its URL here.
Not sure what Google can see?
The Website & SEO Audit checks robots.txt, your sitemap and Search Console errors together.