Free tool
robots.txt generator.
A misconfigured robots.txt lets Google index thousands of junk pages. Set yours up cleanly, with the right blocks.
What's it for?
The robots.txt is the road sign at the entrance to your site: it tells search engines what to crawl and what to ignore. Especially useful on a store, where internal search and filters create thousands of useless addresses.
- Stops Google from wandering onto pointless pages.
- Focuses your ranking on your real pages.
- Declares your sitemap so you're found faster.
Your robots.txt
Place it at the root of your site: example.com/robots.txt. Blocking crawl isn't protection: it doesn't prevent access, it guides robots.