How to Stop Scrapers in 2026: Block AI Crawlers Without Losing Google

Scrapers and AI crawlers cost you bandwidth, compute, AI spend and data. How to detect and block them by user agent and behavior without blocking Google.

Nov 17, 2026路16 min read

Scrapers used to be a nuisance: a competitor copying prices, an SEO tool mapping your links. In the AI age they're a line item. AI companies crawl the web to train models and answer questions with your content. Search-for-AI startups fetch every page you publish. Headless browsers built for AI agents walk your site page by page. Each request is served by your servers, rendered by your framework, sometimes run through your database or your own LLM calls, and you pay for all of it.

This guide covers what scrapers cost you, why blocking them is worth it now, and how to stop them without the classic mistake of blocking Google, your link previews or your own tools. It ends with how SecureNow detects and blocks scrapers automatically, from the dashboard and from the CLI.

The short version:

  1. Sort bots into categories before blocking anything: search engines, link previews and AI assistants stay allowed; AI training crawlers, SEO crawlers and scraping tools are fair game.
  2. Block self-declared scrapers by user agent, not by IP: they rotate addresses faster than any blocklist.
  3. Catch scrapers that pretend to be browsers by behavior: page volume, user-agent rotation, missing browser headers.
  4. Verify anything that claims to be Googlebot.
  5. Block addresses only when they persist, let the blocks expire, and never block your own infrastructure.
  6. Watch what gets stopped, and be able to undo any block in one click.

What counts as a scraper in 2026

"Bot" is too broad to act on. Some bots are how people find you; others only take. Here's the split we use, with real examples from the user agents we see in production:

CategoryExamplesWhat they doBlock?
Search enginesGooglebot, Bingbot, Applebot, DuckDuckBot, YandexBotIndex pages for search results that send visitorsNever
Link previewsfacebookexternalhit, Twitterbot, LinkedInBot, Slackbot, Discordbot, WhatsAppBuild the card when someone shares your linkNever
AI assistantsChatGPT-User, Claude-User, Perplexity-UserFetch a page because a person asked an AI about itNever
Your own toolsAhrefs Site Audit, Screaming Frog, Lighthouse, UptimeRobot, ChecklyAudits and monitoring you set upNever
AI search crawlersOAI-SearchBot, PerplexityBot, Claude-SearchBot, YouBotIndex pages for AI search answersYour call
AI training crawlersGPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, AmazonbotCopy pages into AI training dataYes
SEO & marketing crawlersAhrefsBot, SemrushBot, MJ12bot, DotBot, DataForSeoBot, BarkrowlerMap your site for other people's SEO toolsUsually
Scraping toolsScrapy, HTTrack, PhantomJS, LightpandaDownload your site, page by pageYes
Stealth scrapersFake Chrome or Safari user agents, fake GooglebotsHide what they areYes

Two things stand out. First, the dangerous mistakes are at the top of the table: blocking a search engine or a link-preview fetcher costs you traffic. Second, the hard cases are at the bottom: a stealth scraper looks like a browser, and the only way to catch it is to watch what it does.

One more trap: plain HTTP libraries such as python-requests, okhttp, axios or Go-http-client are not scrapers on their own. They're also what mobile apps, webhooks and server-to-server calls send. Blocking them by user agent breaks your own integrations.

The real impact of scrapers (and what they cost)

Over seven days in November 2026, the sites we run on SecureNow received about 365,000 requests from 35 distinct scrapers, from roughly 5,600 addresses. AhrefsBot alone sent about 168,000 requests. None of it was a visitor.

Here's where that traffic turns into cost.

Bandwidth and hosting bills

Every page a scraper downloads is served and paid for by you: egress from your cloud provider or CDN, plus the compute to produce it. For a static marketing site the egress line is small. It stops being small when crawlers fetch large pages, images or downloads at scale, or when one aggressive campaign runs for days. In our $3,400 egress post-mortem, a scraping campaign ran for 72 hours before anyone noticed.

A quick way to size it for your own app: scraper requests per month 脳 average response size 脳 your egress price per GB, plus the server time of each render. If your pages are server-rendered, the compute usually costs more than the bandwidth.

Your content trains someone else's AI

AI training crawlers copy your pages into datasets used to train models and answer questions. You get no credit, no traffic and no say in how it's used. If your content is your product (docs, articles, listings, reviews, prices), that's your work leaving for free.

AI and API costs

If a page or endpoint calls an LLM (an AI summary on a product page, a chat widget, semantic search, generated content), every bot request runs up your AI bill. A crawler doesn't care that a page costs you a few cents to render. Thousands of pages a day turn that into real money, on top of any third-party API you pay per call.

Database and storage bloat

Bots trigger searches, sessions, analytics events, logs and cache entries. Search queries get stored, carts and sessions get created, logs grow, caches fill with pages no human will request. Your database slows down for the people who matter.

Competitors and data brokers

Prices, product listings, documentation and contact data get harvested and resold. Some of this traffic doesn't come from AI at all: it's the oldest kind of scraping, and it now hides behind browser user agents and residential proxies.

Slower site, skewed analytics

Crawler bursts compete with real visitors for CPU and database connections. They also pollute page views, conversion rates and A/B tests: decisions get made on numbers that include bots.

Why you should block scrapers in the AI age

Blocking used to be risky because the tools were blunt: a WAF that took out Googlebot along with the scrapers cost more than it saved. Three things changed:

  • The bots now say what they are. Most AI and SEO crawlers send a self-declared user agent with a product token (GPTBot, Bytespider, AhrefsBot...). You can block a category precisely, without touching search engines.
  • Search and training are different crawlers. You can stay in Google, Bing and AI search answers while refusing to feed training datasets. Robots.txt tokens such as Google-Extended and Applebot-Extended let you opt out of AI training from the same companies that crawl you for search.
  • The cost moved to you. Server-side rendering, per-request AI calls and usage-based hosting mean every bot request has a price.

robots.txt is still worth having: it states your policy, and well-behaved crawlers respect it. But it's a request, not an enforcement. Scrapers that matter most never read it.

Why robots.txt, rate limits and IP blocklists aren't enough

Most "how to block scrapers" advice stops at one of these. Each has a gap:

  • robots.txt is voluntary. Stealth scrapers ignore it by definition.
  • Per-IP rate limits miss fleets that rotate addresses. One AI search crawler we measured averaged about 2.4 requests per address: no rate limit ever triggers. We covered this in detail in why per-IP rate limits fail against rotating IPs.
  • IP blocklists can't keep up for the same reason, and they're dangerous when they're wrong: block a carrier NAT or a corporate proxy and you block thousands of real people. Real shoppers behind one mobile-carrier address loaded 266 distinct pages in 10 minutes on a store we run: an "obvious scraper" to a naive rule.
  • User-agent matching alone gets fooled both ways. A scraper can claim to be Googlebot, and a careless pattern can match the wrong bot: Claude-SearchBot's user agent contains claudebot@anthropic.com, so a substring match for "claudebot" would block Anthropic's search crawler along with its training crawler.
  • Edge bot management (CDN or WAF products) works, but needs DNS changes, an edge account and someone to tune it. Many small teams never get there.

What works is a combination: precise user-agent rules for bots that name themselves, behavioral detection for bots that don't, verification for bots that claim to be search engines, and IP blocks only for addresses that keep coming back.

How to stop scrapers: a layered approach

You can build this yourself. Here's the recipe, in order:

  1. Recognize the bots you must never block first. Search engines, link previews, AI assistants and your own tools are checked before any scraper rule. Anything that matches one of them is out of scope.
  2. Block self-declared scrapers by product token. Match GPTBot/, Bytespider, AhrefsBot/ and similar as tokens, not substrings, and refuse the request in your app whatever address it comes from.
  3. Detect stealth scrapers by behavior. Per address and per few minutes: how many distinct content pages (ignore APIs, assets, health checks and framework prefetches), how many different browser user agents, and whether real browser headers such as Accept-Language are missing.
  4. Verify search-engine claims. A request that says "Googlebot" from an address Google doesn't publish, or that fails reverse DNS, is not Googlebot.
  5. Block addresses with care. Only addresses that persist, never addresses that also send normal browser traffic, never shared egress (VPNs, private relays, CDNs), always with an expiry.
  6. Make every decision visible and reversible. If you can't see what was blocked and why, you'll eventually block a customer and not know it.

The rest of this guide shows how SecureNow does all six automatically. The framework guides cover the manual side if you'd rather start there: Next.js, Express, Fastify, NestJS and Nuxt.

How SecureNow detects scrapers automatically

SecureNow's Scraper Shield runs on the request traces your app already sends. There's no rule to write and no separate agent to install. Here's what happens behind the dashboard.

1. Every request is classified

Every 10 minutes, Scraper Shield reads the latest window of each app's production traffic and classifies every user agent, in this order:

  1. Never blocked: more than 100 search engines, link-preview fetchers, AI assistants, site-audit and monitoring tools, and HTTP clients used by apps, webhooks and SDKs.
  2. Scraper signatures: more than 80 self-declared crawlers in four categories: AI training, AI search, SEO & marketing, and scraping tools. Patterns match product tokens, never contact e-mails or version numbers.
  3. Everything else is a browser, an HTTP client, or an unknown bot. Unknown bots are recorded but never blocked.

The order matters: a composite user agent that contains both a link-preview token and a scraper token is treated as a link preview.

2. Stealth scrapers are caught by behavior

For traffic with a plain browser user agent, Scraper Shield looks at each address in 10-minute buckets, counting content pages only (APIs, assets, health checks and Next.js prefetches are left out). An address is a stealth scraper when every user agent it sends is a plain browser and at least one of these holds:

SignalThresholdWhy it's safe
Volume600+ distinct pages in 10 minutes, mostly distinct per loadThe busiest real address we measured (a mobile-carrier NAT) loaded 266
User-agent rotation50+ browser user agents, each used at most 3 times, 100+ distinct pagesPeople use one or two browsers; scrapers rotate per request
Missing Accept-Language100+ pages with no Accept-Language at all (on apps that record the header)Every real browser sends it; the busiest person we saw loaded 43 pages

The thresholds were calibrated on real production traffic, with at least a 2x margin over the busiest real person or shared address we observed.

3. Fake search engines are verified

When a request claims to be Googlebot, Bingbot, Applebot, GPTBot or PerplexityBot, its address is checked against the operator's published IP ranges or reverse DNS. An address that fails, or that claims six different companies' crawlers in a day, is flagged as an impersonator.

4. Single-IP and multi-IP scrapers

Scrapers are grouped by identity, not by address. A self-declared crawler seen from many addresses is one multi-IP scraper (a fleet). A stealth scraper on one address is a single-IP scraper. On our own sites, 27 of the 35 scrapers in a week were fleets, which is why address-based blocking alone fails.

5. How blocking works

  • By user agent: self-declared scrapers in a blocked category are refused with a 403 inside your app, whatever address they come from. This needs securenow 8.27.0 or later in your app. Search engines, link previews, AI assistants and site tools are checked first, so they always pass.
  • By address: stealth scrapers, impersonators and some self-declared crawlers are blocked by IP, but only when the address persists (5+ requests over at least 15 minutes in a day) and never sent normal browser traffic. Blocks expire 7 days after the address was last seen.
  • In your app: both run in SecureNow's in-app firewall: no proxy, no DNS change, and a failure to reach SecureNow never blocks traffic.

6. Guardrails against false positives

A scraper blocker that blocks a customer is worse than none. Scraper Shield never blocks:

  • private, loopback and reserved addresses;
  • any address your account authenticates from: your servers' SDK and API-key calls, your CLI and dashboard sessions;
  • your trusted and allowlisted addresses;
  • addresses inside a verified search engine's or AI provider's published ranges (their crawlers are handled by user agent instead);
  • shared egress such as Cloudflare and WARP, where many people sit behind one address;
  • addresses that also serve link previews or ad reviews (Meta's ads crawler, for example, sent about 66,000 requests in two days to a store we run, and blocking it would break your ads);
  • anything from an app whose client addresses can't be verified, such as an app behind a proxy it doesn't trust. Blocking a proxy's address would block everyone behind it.

HeadlessChrome is detect-only: Bing renders pages with it from Microsoft's network (about 1,270 addresses in two days in our data), so blocking it would hurt your search presence.

7. Shared intelligence across every SecureNow app

Scrapers hit many sites at once, so Scraper Shield learns across the network without sharing anyone's data. An address is shared with other accounts only after at least two accounts each saw it scrape persistently and none saw people behind it. New bot names seen across customers are reviewed and, once confirmed, become signatures every SecureNow app blocks. No paths, hosts or account details are shared, only bot names and addresses.

Stop scrapers from the dashboard

Scraper Shield is on for every app once it sends traffic. Here's where it shows up.

The Overview. A Scrapers card sits right under your attack summary: scrapers detected and blocked, requests stopped, single-IP versus multi-IP, a breakdown by category, your top scrapers, and a short reminder of why they matter. Scraper traffic is kept out of your attack numbers: a crawler is a cost, not an attack.

The Scrapers page. One place to see who's blocked and who will keep being blocked:

  • Category switches. Turn blocking on or off for AI training, AI search, SEO & marketing, scraping tools and stealth scrapers. AI search crawlers start as allowed; everything else starts blocked.
  • A switch per scraper. Allow one crawler you rely on (say, the SEO tool your team uses) while its category stays blocked, or block one AI search crawler while the others stay allowed.
  • Multi-IP and single-IP sections. Fleets and lone addresses are listed separately.
  • Every user agent and address. Expand a row to see each user agent with its request count, the addresses (and whether each is blocked now), sample paths, the apps it hit, first and last seen, who operates it, and for behavioral detections, exactly why it was flagged.
  • Never blocked. The search engines, link previews, AI assistants and tools that are always allowed, so you can see what's safe.
  • Other bots we saw. Self-declared bots that aren't in the catalog yet. They're recorded, not blocked.
  • Health and SDK notices. When detection last ran for each app, and which apps need securenow 8.27.0 or later for user-agent blocking.

Only a signed-in person can turn blocking on. API keys and coding agents can turn it off, never on. Unblocking an address anywhere (the Scrapers page, the Blocked IPs page, the attacker page or the CLI) is remembered, so the next detection run doesn't put it back.

Stop scrapers from the CLI

If you live in the terminal, or let your coding agent do the work, the SecureNow CLI has the same data and switches as the Scrapers page with securenow scrapers, plus the firewall commands that show what was stopped.

Set up the SDK and firewall. User-agent blocking needs securenow 8.27.0 or later in your app, and the scrapers command comes with 8.28.0:

npm install securenow@latest
npx securenow init
npx securenow login
npx securenow firewall status

init wires instrumentation and runtime credentials into your project; the Next.js and Express guides cover the details for each framework.

See who's scraping you. One row per scraper: its category, one IP or a fleet, requests, requests stopped, status and id. Filter by category, status or kind, and narrow to one app with --app:

npx securenow scrapers --days 7
npx securenow scrapers --days 30 --kind multi --status blocked

A row that says needs SDK 8.27+ means one of your apps runs a version too old to refuse that scraper by user agent yet.

Find out why one was flagged. show prints the detection reason, every user agent and address with its request count, sample paths and the apps it hit. Ids come from the list (sig:gptbot, ip:<address>, fleet:<id>), and a bare name or address works too:

npx securenow scrapers show gptbot

Block or allow one scraper, or a whole category:

npx securenow scrapers allow sig:ahrefsbot --note "our SEO team uses Ahrefs"
npx securenow scrapers block sig:perplexitybot
npx securenow scrapers policy --category ai_search --action block

An allow is remembered, exactly like a switch on the Scrapers page, so the next detection run doesn't block it again. Turning blocking on takes a person signed in with securenow login; an API key or an agent can only turn it off. npx securenow scrapers rules lists the categories, signatures, behavioral rules and everything that's never blocked.

See what the firewall stopped. Scraper user-agent blocks show up with their rule (for example sig:bytespider), and address blocks show up by IP. Address blocks created by Scraper Shield are also in securenow blocklist list with the source scraper-shield, and unblocking one there is remembered too:

npx securenow firewall hits --since 24h
npx securenow firewall hits --action blocked --since 7d

Exempt your own crawlers and test machines (your servers and sessions are already exempt automatically):

npx securenow trusted add 203.0.113.10 --label "office test runner"
npx securenow firewall test-ip 203.0.113.10

Let your coding agent do it. An agent with a terminal can run every command above. npx securenow mcp also gives Claude Code, Codex or Cursor the firewall and blocklist tools, so you can ask "what did the firewall stop today, and is any of it ours?" from your editor. See blocking attackers from your coding agent over MCP.

Alerts: what you're told, and why scrapers don't page you

A scraper isn't an incident. If every crawler address opened an alert, you'd get hundreds a day, each with an investigation attached, and your alert queue would bury the real attacks. So Scraper Shield doesn't raise an alert per scraper. Instead:

  • The daily digest includes how many scraper requests were stopped and how many addresses are blocked, separately from attacks.
  • The dashboard tells you when something needs you: a detection run that failed for an app, an app whose client addresses can't be verified (so it never blocks by address), or apps that need an SDK update for user-agent blocking.
  • Attack alerts stay clean. Scraper Shield blocks never hide or replace other blocks: if a "scraper" turns out to be probing for vulnerabilities, your attack detections still fire, and a permanent block from you or the agent takes precedence.

If you do want to be paged about scraping, say a burst on a page that calls your LLM, create your own alert rule in plain English:

npx securenow alerts rules from-text "alert me when one address loads more than 300 distinct product pages in 10 minutes" --app <app-key> --periodic

The CLI shows you the checked draft and creates nothing until you confirm. New rules start in test mode, so you can see what a rule would have caught before it notifies anyone. Creating custom alert rules from the CLI and Academy episode 11 walk through testing and promoting it.

Should you block AI search crawlers?

This is the one real business decision, so Scraper Shield leaves AI search crawlers allowed by default:

Block itKeep it
AI training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider...)Your content stays out of training data; less loadYou want your content in future models
AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot...)Less load; your pages don't feed AI answersYour pages can be cited in ChatGPT, Perplexity and Claude answers
SEO crawlers (AhrefsBot, SemrushBot, MJ12bot...)Competitors' tools see less of your siteYour team relies on that tool's data about your site

If AI search sends you traffic or leads, keep it. If your content is the product and AI answers replace visits, block it. Either way it's one switch, and you can reverse it.

A checklist to stop scrapers this week

  1. Add a robots.txt that states your policy, including Google-Extended and Applebot-Extended if you opt out of AI training.
  2. Install or update the SDK to securenow 8.27.0 or later so self-declared scrapers are refused by user agent.
  3. Open the Scrapers page (or run securenow scrapers policy) and check the category defaults: block AI training, SEO and scraping tools; decide on AI search.
  4. Allow any crawler your team depends on with its own switch, or securenow scrapers allow <id>.
  5. Add trusted addresses for test runners or crawlers you operate from machines that never sign in to SecureNow.
  6. After a day, run securenow scrapers --days 7 and securenow firewall hits to see who's scraping you and what's being stopped.
  7. Add a custom alert rule only for pages where a scraping burst costs real money.

For traffic that isn't scraping (credential stuffing, API abuse, vulnerability scanning), the same firewall applies rate limits and instant IP blocks without a deploy. And if you're not sure what you're seeing, start with how to tell if your web app is under attack.

Scraper Shield is included in every SecureNow plan: a 14-day trial of every feature with no credit card, then $99 per app per month.

Frequently Asked Questions

Will blocking AI crawlers hurt my Google rankings?

No, as long as you only block crawlers that aren't search engines. Googlebot, Bingbot, Applebot, DuckDuckBot and other search engine crawlers are a different category from AI training crawlers such as GPTBot, ClaudeBot, CCBot or Bytespider. SecureNow never blocks search engines, link previews or AI assistants fetching a page for a person.

Isn't robots.txt enough to stop scrapers?

No. robots.txt is a request, not a control. Well-behaved crawlers read it; scrapers that hide behind a normal browser user agent never do. Use robots.txt to state your policy (including tokens such as Google-Extended and Applebot-Extended, which opt you out of AI training without leaving search), and enforce it in your app.

Should I block GPTBot and ClaudeBot?

Block AI training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent...) unless you want your content in AI training sets. Think twice about AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot): blocking them removes your pages from ChatGPT and Perplexity answers. Don't block AI assistants (ChatGPT-User, Claude-User, Perplexity-User): a person asked an AI to read your page.

How do I stop scrapers that rotate IP addresses?

Stop treating them as addresses. Self-declared scrapers are blocked by user agent, whatever address they come from. Scrapers that pretend to be browsers are caught by behavior: hundreds of distinct pages in minutes, a new browser user agent on almost every request, or never sending the Accept-Language header real browsers send. IP blocks only make sense for addresses that keep coming back.

Will SecureNow block my own monitoring, SEO audits or Playwright tests?

Site audit and monitoring tools (Ahrefs Site Audit, Semrush Site Audit, Screaming Frog, Lighthouse, UptimeRobot, Checkly...) are never blocked. HeadlessChrome is detect-only because Bing renders pages with it. Every address your servers, CLI or dashboard sessions authenticate from is exempt, and you can add more with securenow trusted add.

How fast does SecureNow block a new scraper?

A crawler that names itself is refused on its first request once your app runs securenow 8.27.0 or later. Scrapers caught by behavior are picked up by the next detection run, every 10 minutes.

Can I see and block scrapers from the terminal?

Yes. securenow scrapers (securenow 8.28.0 or later) lists every crawler and scraper that reached your apps with its category, addresses and status. securenow scrapers show <id> explains why one was flagged, securenow scrapers block and allow switch one scraper, and securenow scrapers policy blocks or monitors a whole category. Turning blocking on takes a person signed in with securenow login.

Do I need Cloudflare or a WAF to block scrapers?

No. SecureNow's firewall runs inside your Node.js app, so there's no DNS change, proxy or edge account. It applies the blocks and user-agent rules your dashboard shows.

How much does SecureNow cost?

Every account starts with a 14-day trial of every feature, with no credit card. After that it's $99 per app per month, Scraper Shield included.

Recommended reading

How to Investigate an Account Takeover by Asking Questions

A step-by-step account takeover investigation in plain English: the ten questions to ask, where each answer lives in your logs, traces and security events, and what to check before you call it closed.

Oct 28
investigate account takeover
Why Per-IP Rate Limits Fail Against Rotating IPs (and What to Count Instead)

One login, 100 rotating IPs: a 10-per-minute-per-IP limit let 275 of 275 requests through. How to spot rotation in your traffic and what to key your limits on.

Oct 27
Rate Limit per API Key, per Session or per IP in Node.js, Without Code Changes

Add rate limits by route and method, counted per IP, per API key, per session cookie or globally, written in plain English and shared across servers with Redis.

Oct 27