Do not block CSS or JavaScript
Search engines need them to render and understand your pages.
Robots.txt Generator
Tell search engines and AI crawlers which parts of your site they may crawl. Block private areas, point crawlers to your sitemap, and decide which AI bots can access your content.
- 100% free
- No sign-up
- Runs in your browser
AI crawlers
Training crawlers collect content to train AI models. Search and answer crawlers fetch pages so assistants can cite and link to you. Blocking those can reduce your visibility in AI answers.
Your code
How to use
How to use the Robots.txt Generator
- Choose the default rule for all crawlers.
- List paths to keep out of search, such as admin or cart pages.
- Add your sitemap URL.
- Decide which AI crawlers to allow or block.
- Download robots.txt and upload it to the root of your domain.
Private by design. Everything is generated in your browser. Your URLs, content and settings are not uploaded or stored.
About this tool
What robots.txt can and cannot do
Robots.txt is a plain text file at the root of your site that gives crawlers instructions about which URLs they may request. Reputable crawlers follow it, which makes it the standard way to keep bots out of admin areas, internal search results and endless filter combinations that waste crawl budget.
It is not a security tool. Blocked pages can still appear in search results if other sites link to them, and badly behaved bots can ignore the file. Use a noindex tag or password protection for anything truly private. For AI, a careful split works well for many businesses: allow the search crawlers that let assistants cite you, and decide separately about training crawlers.
Good to know
Robots.txt tips
Never block your whole live site
Disallow: / on a live site removes it from search over time.
Test before publishing
Use the robots.txt report in Google Search Console after uploading.
Check crawler names
AI companies add and rename crawlers. Review their documentation occasionally.
FAQ
Robots.txt Generator FAQs
Where do I put robots.txt?
At the root of your domain, so it opens at https://www.yoursite.com/robots.txt.
Should I block AI crawlers?
It depends on your goals. Blocking search and answer crawlers can reduce how often AI assistants mention and link to you. Blocking training crawlers keeps content out of future training data.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay, though some other search engines respect it.
Will robots.txt remove a page from Google?
Not reliably. Use a noindex tag on the page instead, and do not block it in robots.txt, so Google can see the noindex.
Want experts to handle your SEO, GEO and AEO?
iCreativez helps businesses rank on Google and get recommended in AI answers.