robots.txt is a plain text file located in the root directory of a website, designed to instruct search engine crawlers (also called web robots or spiders) which pages they can crawl and which they should avoid. It follows the Robots Exclusion Protocol standard and serves as a voluntary agreement between websites and search engines.
While robots.txt has no enforcement power over malicious crawlers, legitimate search engines like Google, Bing, and Baidu respect its directives. Proper use of robots.txt can protect sensitive information, optimize crawl efficiency, and prevent duplicate content from being indexed.
User-agent: Specifies which search engine crawler the rules apply to. Use * for all crawlers, or target specific ones like Googlebot or Bingbot.
Disallow: Prevents crawlers from accessing the specified path. For example, Disallow: /admin/ blocks all content under /admin/.
Allow: Permits crawlers to access the specified path. Useful for allowing specific files within a disallowed directory.
Sitemap: Declares the XML sitemap URL, helping search engines discover and index your content faster.
Crawl-delay: Sets the delay between successive crawler requests in seconds. Note that Google does not recommend this directive.
This robots.txt generator is easy to use. Just follow these steps to create a standards-compliant robots.txt file:
Step 1: Choose a preset template. The tool provides preset templates for WordPress, Shopify, Next.js, Vue, React, and other popular platforms. Click the corresponding button to quickly load recommended rules. For example, selecting the WordPress template automatically blocks sensitive directories like /wp-admin/ and /wp-includes/ while allowing uploads and media files.
Step 2: Customize rules. If the preset template does not fully meet your needs, you can manually add, delete, or modify rules. Each rule consists of an Allow/Disallow type and a path. Click "Add Rule" to add new rules, or use the "Remove" button to delete unwanted ones. Paths support wildcards (e.g., /*.pdf$ matches all PDF files).
Step 3: Configure Sitemap. Enter your XML sitemap URL in the "Sitemap URL" field (e.g., https://example.com/sitemap.xml). This helps search engines discover all pages on your site faster. Most site-building tools (like WordPress with Yoast SEO) auto-generate sitemaps.
Step 4: Preview and Export. All settings are reflected in real-time in the preview panel on the right. Once satisfied, click "Copy Content" to copy the robots.txt content to your clipboard, or click "Download" to save the file directly. Finally, upload the file to your website's root directory to make it active.
Protect Admin Panels: Almost every website has an admin backend (e.g., /admin/, /wp-admin/, /dashboard/). By blocking these in robots.txt, you prevent search engines from indexing login pages, reducing the risk of brute-force attacks and keeping sensitive URLs out of search results.
Prevent Duplicate Content: E-commerce filter pages (e.g., ?sort=price&page=2) and blog tag/category pages often generate massive amounts of near-duplicate content. These are detrimental to SEO. robots.txt can block them, ensuring search engines only crawl your core pages and improving your site's overall quality score.
Control Crawl Frequency: For sites with infrequent updates (e.g., corporate websites, portfolios), Crawl-delay can reduce server load. For large sites, a well-configured robots.txt ensures search engines prioritize high-value pages over low-priority ones.
Multi-language/Multi-region Sites: For websites with multiple language or regional versions, robots.txt can precisely control which content each search engine crawls. For example, you can use different User-agents for Googlebot (global) and Baidu Spider (China), setting different crawl rules for each.
robots.txt vs meta robots: robots.txt controls crawling at the URL level, while meta robots tags (e.g., <meta name="robots" content="noindex">) control indexing at the page level. The key difference: pages blocked by robots.txt may still be indexed by search engines (just not crawled), while meta noindex explicitly tells search engines not to index the page. For content you truly want hidden from search results, use both together.
The Robots Exclusion Protocol Standard: Proposed by Martijn Koster in 1994 and adopted by major search engines in 1997. Google made it an official standard in 2020. The standard defines the syntax and behavior of core directives like User-agent, Disallow, Allow, Sitemap, and Crawl-delay. Although called a "protocol," it is not a legally binding standard but an industry convention that search engines voluntarily follow.
Common Myths: 1) Thinking robots.txt prevents all access—it only affects search engine crawlers, not regular users or malicious bots; 2) Using robots.txt to hide sensitive information—sensitive content should use server-side access controls; 3) Ignoring case sensitivity—while most search engines are case-insensitive, the spec recommends consistent casing; 4) Wrong path format—paths must start with / and use relative paths from the domain root.
robots.txt is a text file placed in the root directory of a website that tells search engine crawlers which pages they can access and which they should not. It follows the Robots Exclusion Protocol standard and serves as a voluntary agreement between websites and search engines.
No. robots.txt only provides suggestions to search engine crawlers, not enforceable access control. Malicious crawlers may ignore robots.txt entirely. To truly block access, use HTTP authentication or server-side access controls.
robots.txt must be placed in the root directory of your website, for example https://example.com/robots.txt. Search engines automatically look for this file at the domain root path.
Adding a Sitemap declaration in robots.txt tells search engines the location of your XML sitemap, helping them discover and index your content more efficiently. Format: Sitemap: https://example.com/sitemap.xml
Yes. robots.txt matches rules from top to bottom, with the first matched rule taking precedence. For example, you can Disallow an entire directory and then Allow specific files within it. However, different search engines may interpret rules with slight differences.
Crawl-delay specifies the delay in seconds between successive crawler requests. However, Google does not recommend using this directive and suggests using the crawl rate setting in Google Search Console instead. Bing and some other search engines support this directive.
Common reasons include: 1) File not in the root directory; 2) Wrong filename (must be robots.txt, not robot.txt); 3) Format errors (e.g., missing User-agent line); 4) Search engine cache (changes may take days to take effect); 5) Unsupported wildcards.
Yes. By specifying different User-agents, you can set different rules for different search engines. For example, User-agent: Googlebot targets Google, and User-agent: Bingbot targets Bing. Use User-agent: * to apply rules to all search engines.