Robots.txt Generator: A robots.txt generator creates a valid robots.txt file that tells search engine and AI crawlers which parts of your site they may access. Choose allow/disallow rules and your sitemap URL, then copy the ready file for your site root.
Frequently Asked Questions
What is a robots.txt file?
robots.txt is a plain-text file in your site's root that gives crawling instructions to bots. It uses User-agent, Allow and Disallow rules to control which paths search engines may crawl.
Does robots.txt block a page from Google?
No. robots.txt controls crawling, not indexing. To keep a page out of search results, use a noindex meta tag instead, since a disallowed page can still be indexed if linked elsewhere.
How do I allow AI crawlers in robots.txt?
Add User-agent lines for bots such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended with Allow: /. This generator can include those AI crawler rules for you.
Where do I put the robots.txt file?
Place it at the root of your domain so it is reachable at https://yoursite.com/robots.txt. Crawlers check this exact location before crawling.