Aggressive crawlers and scrapers, AI bots included: throttle or ban?

Not every flood is an attack. A search engine re-crawling your catalogue, an SEO tool auditing a competitor, an AI company collecting training data, a price-comparison scraper running every hour: none of them wants your server down. Some of them can still bring it down, especially when they walk into a part of your site with an infinite number of URLs.

This article is about traffic that is unwanted in volume rather than in intent — and why the right answer is usually to slow it down, not to shut it out.

The names

• Crawlers or spiders: Googlebot, Bingbot and the other search engines; SEO tools such as AhrefsBot, SemrushBot, MJ12bot; archive crawlers.
• AI crawlers, which collect pages to train or ground language models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, whose data many models are trained on), Bytespider (ByteDance), PerplexityBot, and others.
• Scrapers: tools copying content, prices or listings, often without identifying themselves.

What they want

Your content. Search engines want to index it and send you visitors; SEO tools want to measure it; AI crawlers want text to train on or to quote; scrapers want specific data. The cost to you is a side effect: they want every page, as fast as they can get it.

How it becomes a problem

Three patterns turn a crawler into an incident:

1. Crawl traps. Faceted navigation — filters, sort orders, pagination, calendars — generates a near-infinite number of distinct URLs: /shop?color=red&size=m&sort=price&page=37. Each one is an uncached database query. A crawler that follows every link will request all of them.
2. Parallelism. A crawler running many connections at once to a site that renders every page through PHP is, from your server’s point of view, an HTTP flood.
3. No identity. Scrapers that use a browser user agent and rotate addresses cannot be told apart from visitors by their headers.

What it looks like

The access log shows a few user agents dominating dynamic requests, deep pagination and query-string combinations no human would produce, and a regular, tireless rhythm:

# Top user agents among requests to PHP pages (combined log format)
awk -F'"' '$2 !~ /\.(css|js|png|jpe?g|gif|svg|webp|woff2?|ico)/ {print $6}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head

A user agent is a claim, not a proof. Anyone can send Googlebot. To verify one, do a reverse DNS lookup on the address and a forward lookup on the result — a real Googlebot resolves to a name under googlebot.com or google.com, which resolves back to the same address:

host 192.0.2.66          # -> crawl-192-0-2-66.googlebot.com (if it really is Google)
host crawl-192-0-2-66.googlebot.com   # must return 192.0.2.66

Google, Bing and OpenAI also publish the address ranges of their crawlers.

How to protect yourself

Tell them what you want: robots.txt

robots.txt (standardised as RFC 9309 in 2022) tells crawlers what not to fetch. Use it to fence off crawl traps and to opt out of crawlers you do not want:

User-agent: *
Disallow: /search
Disallow: /*?*sort=
Disallow: /cart

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Its limits matter. It is a request, not an enforcement: well-behaved crawlers, including the major search engines and the AI crawlers that publish their user agents, say they honour it; scrapers do not read it at all. Crawl-delay is honoured by some crawlers, such as Bing, and ignored by Google. And Google-Extended is a robots.txt token controlling whether Google may use your content for its AI models — not a separate crawler you will find in your logs.

Fix the trap

The best defence against crawl traps is not serving them: add rel="nofollow" to filter links, mark filtered pages noindex and point canonicals to the unfiltered page, and cache what you can. A crawler that cannot find infinite URLs cannot request them.

Throttle by behaviour, per address and per bot

A rate limit slows a crawler without losing the search traffic it brings. Answer excess requests with 429 Too Many Requests, which well-behaved crawlers understand as “back off”. In nginx you can give known bots a stricter budget than browsers by keying the limit on a map:

map $http_user_agent $bot_key {
    default                 "";
    ~*(bot|crawler|spider)  $binary_remote_addr;
}
limit_req_zone $bot_key zone=bots:10m rate=1r/s;

server {
    location ~ \.php$ {
        limit_req zone=bots burst=10;
        limit_req_status 429;
    }
}

Requests with an empty key are not limited, so browsers pass untouched. Combine it with the per-address limit from the HTTP flood article for clients that do not announce themselves.

Ban only what you never want

Blocking by user agent (return 403 on a match) suits crawlers you never want and that identify honestly. Blocking by address is a blunt tool for scrapers: they rotate, and the address you ban today may be a customer’s mobile network tomorrow. Keep bans for behaviour that is unambiguous, and use a CDN’s bot management when the scraping is distributed.

How Demerzel helps

• It shows who loads your servers. With web logs granted, each server’s Security tab ranks the busiest and the most error-prone addresses, and each address’s dossier shows its user agents, sampled paths (query strings removed), the sites it hit and its status codes — on all your servers at once. The timeline lines that traffic up with the 5xx your sites returned and the machine’s load, so you can tell a crawler that was merely busy from one that hurt.
• It throttles instead of banning. The rate-dynamic rule fires on more than 300 dynamic requests in two minutes from one address across your sites; static assets do not count. When the warden enforces it — after a replay on your own logs showed it would not have caught a real visitor — it does not cut the address off: it limits the address’s new connections to your web ports, two a second by default. That is what slows an aggressive crawler without blocking a heavy but legitimate client.
• It will not cut off a big share of your traffic by itself. An address that carried more than 10% of the server’s successful requests over the last hour is never blocked by a rule — an office NAT, a mobile carrier, a partner’s integration. Demerzel raises an Action, and you decide: allow it with a note, or block it yourself. Cloudflare addresses are never blocked at all, since every visitor behind them would be.

What it does not do: judge user agents. Demerzel’s rules look at what an address does — its rate, its errors, the sites it visits, the paths it asks for — not at what it claims to be. It does not verify Googlebot, keep a list of AI crawlers, or read robots.txt. Those decisions stay yours, in robots.txt and your web server configuration.