What is the difference between Disallow: / and Disallow: ?

Disallow: / (with a slash) tells bots not to crawl any page on the entire site, effectively blocking your whole website. Disallow: (with nothing after it) means nothing is disallowed, effectively allowing bots to crawl everything.

Back to Toolbox

Robots.txt Generator

Q: What is a robots.txt file?

A robots.txt file is a plain text file that tells search engine crawlers (like Googlebot) which pages or files the crawler can or cannot request from your site. It acts as a guide, not an unbreakable command, but all reputable bots respect its rules.

Q: Where do I put the robots.txt file?

The file must be named exactly robots.txt and placed in the root directory of your website. For example, for www.seogeo.co, the file should be accessible at https://www.seogeo.co/robots.txt.

Q: What is a User-agent?

A User-agent is the specific name of the crawler or bot you want to give instructions to. User-agent: * is a wildcard that applies to all bots. User-agent: Googlebot applies only to Google's main crawler.

Create a well-formed robots.txt file to guide search engines and AI bots on how to crawl your website effectively.

Default Policy for All Crawlers

Set the default access policy for all user-agents (*).

Sitemap URL (Recommended)

Include the full URL to your XML sitemap.

Custom Rules for Specific Bots

Add specific Disallow rules for individual crawlers.

Generated robots.txt

# Your generated robots.txt file will appear here.

Copy the generated text and save it as a robots.txt file in your root directory.

How to Use This Tool

Set Default Policy

Choose whether to allow or disallow all search engines (User-agent: *) by default. "Allow All" is recommended for most websites.

Add Sitemap URL

Paste the full URL to your sitemap.xml file. This is highly recommended as it helps search engines discover all your important pages.

Add Specific Rules

If you need to block specific bots from certain sections, click "Add New Rule". Select the bot and enter the path you want to disallow (e.g., /admin/).

Generate & Copy

The robots.txt content is generated automatically on the right. Once you're happy with the rules, click the "Copy" button.

Upload to Your Site

Create a new file named exactly robots.txt (all lowercase). Paste the copied content into this file and upload it to the root directory of your website.

Frequently Asked Questions

What is a robots.txt file? +

A robots.txt file is a plain text file that tells search engine crawlers (like Googlebot) which pages or files the crawler can or can't request from your site. It acts as a guide, not an unbreakable command, but all reputable bots respect its rules.

Where do I put the robots.txt file? +

The file must be named robots.txt and placed in the root directory of your website's host. For example, for www.seogeo.co, the file should be accessible at https://www.seogeo.co/robots.txt. It will not work in any other location.

What is a User-agent? +

A User-agent is the specific name of the crawler or bot you want to give instructions to. User-agent: * is a wildcard that applies to all bots. User-agent: Googlebot applies only to Google's main crawler. This allows you to set different rules for different search engines.

What's the difference between Disallow: / and Disallow: ? +

This is a critical distinction.
• Disallow: / (with a slash) tells bots not to crawl any page on the entire site. It effectively blocks your whole website.
• Disallow: (with nothing after it) means nothing is disallowed, effectively allowing bots to crawl everything. This is the correct rule to use for "Allow All".