Robots.txt Generator
Configure which search engine bots can access which parts of your site, then generate and download your robots.txt file.
Reviewed by the ToolNestr Editorial Team — July 2026
How robots.txt directives work
The Robots Exclusion Standard uses a plain-text format where each directive is on its own line. A User-agent line targets a specific crawler (or all crawlers with *). The directives that follow apply to that user-agent until the next User-agent line. Disallow blocks a path, Allow overrides a block for a sub-path, and Sitemap points to your XML sitemap. Crawlers read this file first and obey its rules before crawling.
Worked example
Given these rules in the generator:
User-agent: Googlebot Disallow: /admin/ Disallow: /private/ Allow: /admin/public/ User-agent: * Disallow: /cgi-bin/ Sitemap: https://example.com/sitemap.xml
This tells Googlebot it can access /admin/public/ but not /admin/ or /private/. All other crawlers are only blocked from /cgi-bin/. The sitemap URL is provided for all crawlers. When Googlebot visits the site, it fetches robots.txt, parses these rules, and skips all disallowed paths during its crawl.
Robots.txt directives usage
How different directives are used across websites.
Web Developer
Configures crawl access during site builds. Blocks staging environments, admin panels and duplicate content paths from being indexed by search engines.
SEO Specialist
Audits robots.txt to ensure critical pages are crawlable. Manages crawl budget by disallowing low-value pages like tag archives and pagination filters.
Site Owner
Runs a personal or small business site. Creates a simple robots.txt from scratch without needing to understand the syntax in detail.
DevOps Engineer
Manages robots.txt across multiple environments (dev, staging, production). Ensures crawlers only access public-facing environments and not internal systems.
Common robots.txt directives
| Directive | Example | Purpose |
|---|---|---|
| User-agent | User-agent: Googlebot | Target a specific crawler |
| Disallow | Disallow: /admin/ | Block a path from crawling |
| Allow | Allow: /public/ | Override a disallow for a sub-path |
| Sitemap | Sitemap: https://example.com/sitemap.xml | Point to your XML sitemap |
| Crawl-delay | Crawl-delay: 10 | Seconds between requests |
| Host | Host: https://example.com | Specify canonical host (Yandex) |
How to use the Robots.txt Generator
Add user-agents
Start with "*" for all crawlers or specify a specific bot like Googlebot. Each user-agent can have its own set of rules.
Add allow/disallow rules
For each user-agent, add paths to allow or disallow. Use the "Add rule" button and select the directive type and path.
Generate and deploy
Click Generate to preview the file. Copy the output or download it, then upload to the root of your website.
Tips for robots.txt management
Don't block CSS, JS or images
Blocking stylesheets, scripts or images prevents search engines from rendering your pages correctly. Google uses rendered content for ranking, so allow these resources.
Use Allow to refine blocks
If you block an entire directory but need one sub-path accessible, use Allow first then Disallow. Order matters — the most specific rule wins.
Test before deploying
Use Google Search Console's robots.txt tester or check the generated output manually. A mistake could accidentally block your entire site from search engines.
What is robots.txt?
Robots.txt is a plain text file that tells search engine crawlers which URLs they can and cannot request. It follows the Robots Exclusion Standard and is the first file crawlers look for when they visit a site. It is a voluntary protocol — well-behaved crawlers obey it, but malicious bots may ignore it.
How search engines use it
When a crawler like Googlebot arrives at a site, it fetches /robots.txt first. It reads the directives, identifies which paths are off-limits, and then crawls only the allowed pages. This helps manage crawl budget by directing bots to high-value pages.
Common directives
The most common directives are User-agent (targets a crawler), Disallow (blocks paths), Allow (permits sub-paths), Sitemap (points to your XML sitemap) and Crawl-delay (rate limits requests). Each directive applies to the most recent User-agent declaration above it.
Testing your robots.txt
Always test your robots.txt after changes. Google Search Console provides a built-in tester. You can also check manually by visiting yourdomain.com/robots.txt in a browser. Common mistakes include syntax errors, missing newlines and accidentally blocking the root path.
Frequently asked questions
What is a robots.txt file?
Robots.txt is a text file placed at the root of a website that tells search engine crawlers which pages or sections they can or cannot access and index.
Where should robots.txt be placed?
The file must be placed in the root directory of your domain, e.g. https://example.com/robots.txt. It is the first file crawlers check when visiting a site.
What is a User-agent directive?
User-agent specifies which crawler the rules apply to. Use "*" to target all crawlers, or specific names like "Googlebot" or "Bingbot" for search-engine-specific rules.
What does Disallow do?
Disallowing a path tells crawlers not to access it. A single slash "Disallow: /" blocks everything. An empty "Disallow:" allows everything for that user-agent.
What does Allow do?
Allow overrides a Disallow rule for a specific path. It is used when you want to block a directory but allow a specific file or subdirectory within it.
What is a Sitemap directive?
The Sitemap directive points crawlers to your XML sitemap location. Adding this helps search engines discover and index your pages more efficiently.
What is Crawl-delay?
Crawl-delay tells crawlers how many seconds to wait between requests. It helps reduce server load but is not supported by all search engines (Googlebot ignores it).
Can robots.txt block sensitive content?
No. Robots.txt is a voluntary protocol. It only prevents well-behaved crawlers from accessing pages. Malicious bots ignore it. Use authentication or noindex for truly sensitive content.
Does robots.txt affect SEO?
Indirectly. Proper robots.txt ensures crawlers find your important pages. Blocking critical pages can prevent them from being indexed. Always test your robots.txt after making changes.
How can I test my robots.txt?
Google Search Console has a robots.txt tester. You can also use the preview in this tool to verify your directives look correct before deploying.