Developer Security Suite

100% Private In-Browser Generation

AI Bot & Scraper Blocker (robots.txt Generator)

Protect your content from unauthorized AI model training while preserving traffic from Google, Bing, and AI search engines.

1-Click Presets

8 / 12 Blocked

These bots scrape your articles, code, and media solely to train proprietary LLM models without providing attribution.

GPTBotOpenAI

Scrapes web content to train OpenAI models (GPT-4/GPT-5).

ClaudeBotAnthropic

Crawls content for training Anthropic Claude models.

Google-ExtendedGoogle

Used for training Google Gemini and Vertex AI models.

BytespiderByteDance

Aggressive ByteDance crawler used for training TikTok AI.

CCBotCommon Crawl

Public web scraper dataset frequently used in training datasets.

Meta-ExternalAgentMeta

Scrapes web data to train Meta LLaMA and AI models.

DiffbotDiffbot

Commercial AI web scraping & knowledge graph crawler.

cohere-aiCohere

Scrapes content to train Cohere enterprise language models.

These bots query your pages in real-time to answer user prompts and cite links directly back to your site.

OAI-SearchBotOpenAI

Used by ChatGPT Search to find real-time answers and cite web sources.

PerplexityBotPerplexity

Crawls content for real-time citations in Perplexity AI search.

Essential web crawlers for organic search visibility on Google and Bing.

GooglebotGoogle

Indexes web pages for Google Web Search results.

BingbotMicrosoft

Indexes web pages for Bing & Yahoo search engines.

Sitemap XML URL (Optional)

Custom Paths to Disallow (Comma-separated)

robots.txt

47 lines•852 chars

# ==============================================================================
# robots.txt generated by DevsLoad AI Bot & Scraper Blocker
# https://devsload.com/ai-robots-txt-generator
# ==============================================================================

# --- Allowed Search Engines & AI Search Assistants ---
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# --- Blocked AI Training Scrapers & Crawlers ---
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: cohere-ai
Disallow: /

# --- General Crawler Rules ---
User-agent: *
Allow: /

Why Use DevsLoad AI Bot Blocker?

Granular AI Differentiation

Distinguishes between training scrapers (GPTBot, ClaudeBot) and search bots (OAI-SearchBot, PerplexityBot) so you never lose search traffic.

Safe Syntax Without Penalties

Guarantees valid robots.txt RFC syntax without accidentally putting Disallow on Googlebot or Bingbot.

Zero Server Telemetry

Generates your configuration 100% in your local browser thread. Your domain and paths are never tracked.

Instant Download & Copy

Export a ready-to-deploy robots.txt file to drop straight into your public web root or static site generator.

How to Protect Your Website from AI Scrapers

1. Choose a Preset

Select "Smart Protection" to automatically disallow LLM training scrapers while keeping citation-friendly search bots allowed.

2. Fine-Tune Bot Toggles

Customize individual crawler toggles (such as Bytespider, CCBot, or Meta-ExternalAgent) according to your privacy policy.

3. Add Sitemap & Deploy

Optionally add your sitemap URL, click "Download robots.txt", and place the file in your website root folder (e.g. yourdomain.com/robots.txt).

Frequently Asked Questions

What is the difference between GPTBot and OAI-SearchBot?

GPTBot is OpenAI's scraper used to gather training data for future generative models. OAI-SearchBot is OpenAI's search crawler used by ChatGPT to search the live web and provide clickable citations back to your website.

Do all AI crawlers respect robots.txt?

Major providers including OpenAI, Anthropic, Google, and Meta have officially committed to respecting robots.txt directives. Blocking them prevents their official user-agents from downloading your pages.

Will blocking AI training scrapers hurt my Google search ranking?

No! Google uses standard Googlebot for search indexing. Disallowing "Google-Extended" only opts you out of Gemini AI training, without affecting your Google search rankings in any way.

Where should I upload the robots.txt file?

Save the generated file as "robots.txt" and upload it to the root directory of your website domain so it is accessible at https://yourdomain.com/robots.txt.

Related Developer Tools

Boost your development workflow with more free, 100% private in-browser tools.