Robots.txt Generator
Build a valid robots.txt in your browser: add per-crawler Allow and Disallow rules, a sitemap and crawl delay, then copy the result and deploy it at your site root.
Crawler rules
Build a robots.txt by adding rules per crawler. Use * to address all bots; rules with the same user-agent are grouped automatically.
Your robots.txt
Save this as robots.txt at the root of your domain (e.g. https://example.eu/robots.txt).
User-agent: *
Disallow: /admin/About Robots.txt Generator
A robots.txt is a plain-text file placed at the root of a domain that tells search-engine crawlers and other automated agents which URL paths they may or may not request. It follows the long-standing Robots Exclusion Protocol and is read from a fixed location - for example https://example.eu/robots.txt - before a compliant bot crawls the rest of the site. It is a set of instructions for well-behaved crawlers, not an access-control or security mechanism.
This generator lets you assemble that file directive by directive. Each rule pairs a User-agent with an Allow or Disallow directive and a path; enter * to address every crawler, or a specific name such as Googlebot or Bingbot to target one. Rules that share the same user-agent are merged into a single group automatically, so the output stays clean and standards-compliant no matter what order you add them in.
Two optional fields round out the file. A Sitemap line points crawlers at your XML sitemap as an absolute URL, helping search engines discover new pages faster. A Crawl-delay sets a pause between requests - a non-standard directive honored by engines like Bing and Yandex but ignored by Google, which manages crawl rate from Search Console instead. As you edit, the finished file is rendered live and ready to copy.
Because the tool runs entirely in your browser, your rules, paths and sitemap URL never leave your device - nothing is uploaded to a server. Once you are happy with the output, save it as a file literally named robots.txt and upload it to the top level of your web root so it resolves at /robots.txt on each host you want to govern.
How to use it
- 1Add a rule: pick a User-agent (* for every bot, or a name like Googlebot), choose Disallow or Allow, and type the path such as /admin/.
- 2Click Add rule to stack more directives - rules that share a user-agent are grouped for you automatically.
- 3Optionally paste your sitemap's absolute URL and set a crawl-delay in seconds.
- 4Copy the generated file and save it as robots.txt at your web root so it is reachable at https://yourdomain/robots.txt.
- 5Confirm it loads in a browser, then test specific paths using the robots.txt report in Google Search Console.
Common use cases
- -Keep staging areas, admin panels, or faceted-search URLs from being crawled by search engines.
- -Point crawlers to your XML sitemap so newly published pages are discovered sooner.
- -Give individual bots - Googlebot, Bingbot, GPTBot - different rules from the catch-all group.
- -Throttle aggressive crawlers on a small server using a crawl-delay.
- -Produce a clean, correct starter robots.txt when launching a new site.
Frequently asked questions
- What is a robots.txt file?
- A robots.txt is a plain-text file placed at the root of a domain (for example https://example.eu/robots.txt) that tells crawlers which URL paths they may or may not request, following the Robots Exclusion Protocol. It guides well-behaved bots but does not restrict access or protect private content.
- Does robots.txt stop a page from being indexed?
- No. Disallow only prevents compliant crawlers from fetching a URL; a blocked page can still appear in search results if other sites link to it. To keep a page out of the index, allow crawling and add a noindex meta tag or X-Robots-Tag header instead.
- What does User-agent: * mean in robots.txt?
- User-agent: * applies the rules beneath it to every crawler that does not have its own named block. You can add specific blocks, such as User-agent: Googlebot, to give particular bots different rules.
- Is the Crawl-delay directive supported by Google?
- No. Crawl-delay is non-standard. Bing and Yandex honor it, but Googlebot ignores it - Google's crawl rate is controlled in Google Search Console instead.
- Where should the robots.txt file be placed?
- At the top level of each host, reachable at /robots.txt, such as https://www.example.eu/robots.txt. Crawlers only read it from the root, so a copy in a subfolder has no effect.
- Is this robots.txt generator private?
- Yes. The file is assembled entirely in your browser with client-side JavaScript, so your rules, paths and sitemap URL are never sent to any server.