Is this robots.txt generator free?

Yes — completely free, no signup. It builds a robots.txt AI-crawler policy entirely in your browser. Copy or download it and add the lines to your existing robots.txt.

Is Google-Extended the same as Googlebot?

No. Googlebot is Google's classic search crawler and blocking it hurts search ranking, so it is deliberately not in this tool. Google-Extended is a separate user-agent for Gemini and AI training that you can block without affecting search results.

Where does the robots.txt file go?

At the root of your domain: https://yourdomain.com/robots.txt. If you already have a robots.txt, append these User-agent blocks to it rather than replacing the file.

Do AI crawlers actually respect robots.txt?

The major ones — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot — publicly commit to honouring robots.txt Disallow rules. Rogue crawlers that ignore it are a separate problem best solved with WAF or firewall rules, not robots.txt.

`robots.txt` AI generator

Decide which AI crawlers can read your site. Toggle GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, add your sitemap, then copy or download a ready-to-ship robots.txt. Runs entirely in your browser — no signup.

Free tool · updated 2026-09-03

Preset:

Sitemap URL (optional)

Your robots.txt (editable)

Audit my site →

How it works

Choose a policy. Start from a preset or toggle each crawler. Allowed emits Allow: /; Blocked emits Disallow: / for that user-agent.
Add your sitemap (optional) so crawlers can discover your pages — one Sitemap: line is added to the file.
Ship it. Copy or download, then add the lines to robots.txt at your domain root. If you already have one, append these blocks rather than replacing the file.

Note: Googlebot is intentionally not in this list — blocking it hurts normal search ranking. Google-Extended is the separate AI user-agent you can block without affecting search. Full background in the robots.txt for AI crawlers guide.

Which crawlers should I allow?

If your goal is to get cited by AI answer engines, allow the answer-engine crawlers (GPTBot, PerplexityBot, ClaudeBot, Google-Extended and friends) — that's how ChatGPT and Perplexity learn your pages exist. If your content is your product and you don't want it used for bulk model training, block dataset crawlers like CCBot and Bytespider. The Recommended preset does exactly this split.

Allowing a crawler in robots.txt is necessary but not sufficient to actually get cited — answer engines also need to understand your pages. Run the free AICite audit to see all 13 signals, or build your llms.txt with the free generator.

FAQ

Is this generator free?

Yes — completely free, no signup, and it runs entirely in your browser. The $24 Pro Report bundles the robots.txt patch with llms.txt, llms-full.txt, JSON-LD, and an AI-Ready badge.

Should I allow or block AI crawlers?

Allow answer engines if you want AI citations; block bulk trainers if your content is a paid product. The worst option is no rule at all — an implicit allow you never chose. The Recommended preset is a safe default for most sites that want traffic and citations.

Do I replace my existing robots.txt?

No — append these User-agent blocks to your current robots.txt. Keep any existing User-agent: * group and your existing Sitemap: line if you already have one.

How do I verify it's working?

Fetch it with a spoofed agent: curl -A "GPTBot" -I https://yourdomain.com/robots.txt. Then run the free AICite audit and check the /robots.txt signal.

Built by AICite — the free A–F audit for how AI answer engines see your site. Configure your policy above, then grade your whole site free.