Writing robots.txt: User-agent groups, Allow/Disallow, and Sitemap tips

robots.txt is a plain-text file at your site root (https://example.com/robots.txt) that tells cooperating crawlers which paths they may fetch. It is advisory, not authentication: polite bots read it; scrapers and misconfigured clients may ignore it. Treat it as crawl guidance and public documentation of intent—not as a way to hide private URLs.

This guide covers User-agent groups, Allow / Disallow, optional Crawl-delay and Host, and Sitemap tips. Draft and preview locally so staging paths and internal prefixes never need to leave your machine—see Why “local only” matters for developer tools.

What a crawler reads

A typical file is a sequence of groups. Each group starts with one or more User-agent: lines, then rule lines that apply to those agents:

User-agent: *
Disallow: /admin/
Allow: /admin/public/

Sitemap: https://example.com/sitemap.xml

Directive	Role
User-agent	Which bot (or * for all) this group targets
Disallow	Paths the agent should not crawl
Allow	Paths that remain allowed (useful exceptions inside a broader disallow)
Sitemap	Absolute URL of a sitemap (often after all groups)
Crawl-delay	Seconds between requests (non-standard; support varies)
Host	Preferred host hint (mostly Yandex; Google largely ignores it)

Blank lines separate groups. Lines starting with # are comments. Path matching is prefix-based on the path portion of the URL (query strings are usually ignored by major engines for matching, but do not rely on robots.txt for access control either way).

User-agent groups

Crawlers pick the most specific matching group for their token when several groups exist. A dedicated User-agent: Googlebot group overrides the catch-all * for Googlebot when both are present. Common patterns:

• One * group — site-wide defaults (allow marketing pages; block admin, staging, or API junk).
• Extra groups for named bots — tighten or loosen rules for Googlebot, Bingbot, or AI crawlers without changing everyone else’s rules.
• Multiple User-agent: lines in one group — same Allow/Disallow set for several named agents (the builder emits one rule block with several agent lines).

Presets are starting points, not gospel. LocalTools ships Allow all, Disallow all, Block AI crawlers, WordPress-style, and Next.js shapes so you can edit paths for your routes. Always skim Disallow lists before production—blocking /_next/static/ or /api/ may be fine for crawl budget, but over-blocking product URLs hurts discovery.

Allow vs Disallow: order and specificity

Both directives take a path prefix (often starting with /). Engines combine them with their own longest-match / most-specific rules; Google documents that the most specific rule wins when Allow and Disallow conflict. Practical habits:

1. Prefer narrow Disallow paths (/admin/, /checkout/internal/) over Disallow: / plus a long Allow whitelist unless you truly want a private or pre-launch site.
2. Use Allow for exceptions under a blocked tree (classic WordPress pattern: disallow /wp-admin/ but allow admin-ajax.php).
3. Remember empty Disallow (or allowing /) means “crawl freely” for that agent group—not “no robots.txt.”
4. robots.txt does not secure content. Anything linked or guessed can still be fetched by non-compliant clients. Use auth, noindex, or network controls for secrets.

Disallow: / blocks the whole site for matching agents. That is useful for staging hosts that should not be indexed, but confirm the file is on the host you intend—production and staging share the same path name.

Sitemap tips

Sitemap: lines tell crawlers where to find your URL list. Tips that avoid common mistakes:

• Use absolute HTTPS URLs (https://example.com/sitemap.xml), not relative paths.
• You may list more than one sitemap (or a sitemap index URL). Keep the list honest: do not point at broken or password-gated XML.
• Sitemaps do not override Disallow. If a URL is disallowed, listing it in a sitemap does not force crawling; fix the conflict deliberately.
• After you generate or change a sitemap, validate structure (urlset vs index, duplicate locs) with a local Sitemap XML formatter before deploying.

robots.txt and sitemaps are complementary: robots.txt steers behavior; sitemaps advertise candidates. Meta tags and structured data in the page HTML are separate layers for how a URL appears in social previews and rich results.

Crawl-delay and Host (optional, uneven support)

Crawl-delay asks some bots to wait N seconds between requests. It was never part of the original exclusion standard; Google ignores it in favor of Search Console crawl settings. Use it sparingly for bots that document support, and prefer CDN / server rate limits for real protection.

Host (without https://) is mainly a Yandex preference for the canonical hostname. For Google, prefer consistent redirects, canonical URLs, and Search Console property setup instead of relying on Host:.

A safe local drafting workflow

1. Decide whether the environment is public production, staging (block all), or mixed (public site + private prefixes).
2. Start from a preset that matches your stack, then edit User-agent lines and paths.
3. Add Sitemap absolute URLs last; leave Host blank unless you know you need Yandex’s directive.
4. Copy or download robots.txt, place it at the site root, and confirm it is reachable over HTTPS.
5. Verify with Search Console / Bing Webmaster (or a crawler tester). Treat the file as public—anyone can fetch it.

Building rules in-tab keeps draft paths and notes on your device. For the privacy model behind local generators, see Why “local only” matters for developer tools.

Try it locally in your browser

Use the robots.txt generator to:

• Start from Allow all, Disallow all, Block AI crawlers, WordPress, or Next.js presets.
• Edit User-agent groups with Allow / Disallow, optional Crawl-delay, Sitemap URLs, and Host.
• Preview, copy, or download robots.txt—generation runs only in this tab; nothing is uploaded to LocalTools.

Review paths before deploying. robots.txt is a public contract with crawlers, not a substitute for access control.

Related reading

• Sitemap XML formatter — pretty-print and audit urlset / index documents you reference from Sitemap:.
• Slug rules for URLs and filenames — consistent public paths that sitemaps and crawlers will see.
• Writing a good .gitignore: presets, negation, and what not to ignore — another “presets then edit” workflow for repo hygiene.
• Why “local only” matters for developer tools — why draft crawl rules belong in-tab, not on a random paste site.