HTTP floods: when every request is valid and there are too many

The slow attacks in this series send almost nothing. An HTTP flood is their opposite: complete, well-formed requests, as many as the attacker can generate, aimed at the pages that cost you the most to build. Nothing about any single request is wrong. There are just too many.

The names

HTTP flood, layer 7 DDoS or application-layer DDoS. Variants are named after what they hit: GET floods, POST floods, cache-busting floods (random query strings so the cache never hits), login floods. When a single scanner or crawler does the same thing without meaning to, the effect on your server is identical — the crawler article covers that case.

What the attacker wants

Your server has to spend real work on each request: a PHP process, a few database queries, perhaps a call to a search backend. The attacker wants that work to exceed what you have, so that legitimate requests queue, time out, or fail with 502 and 504. The goal is usually extortion, protest or competition; sometimes it is cover for something else happening at the same moment.

The asymmetry is what makes it work. A request costs the attacker a few hundred bytes. An uncached search page can cost you 200 ms of CPU and a database connection.

How it works

Floods come from three kinds of sources, each harder to block than the last:

1. One machine, or a handful — a rented server running a load tool. Easy to see and to ban.
2. A botnet of compromised devices and servers, each sending a moderate rate. Hundreds or thousands of addresses.
3. Residential proxies — real home connections rented by the hour. Each address sends a few requests, from the same ISPs your customers use.

The requests target what is expensive and cannot be cached: search (/?s=, /search?q=), the cart and checkout, login pages, wp-admin/admin-ajax.php, API endpoints, or any URL with a random query string appended (/?a8f3c=1) to bypass caches. More sophisticated tools rotate user agents, send realistic headers and even complete TLS fingerprints of real browsers.

What it looks like on your server

Unlike a slow attack, a flood is loud in every metric:

• The access log grows fast. Requests per minute several times the usual, concentrated on a few paths.
• Load and CPU climb. PHP-FPM workers are all busy; the load average goes well above the number of vCPUs.
• PHP-FPM hits its ceiling. Its log says WARNING: [pool www] server reached pm.max_children setting (20), consider raising it.
• The database follows. Too many connections errors from MySQL if pm.max_children across all pools is larger than max_connections.
• Errors reach visitors. 502 and 504 from the web server when PHP-FPM cannot accept more work.

To see who is behind it, rank the addresses in the last few thousand log lines (combined log format, client address first):

tail -n 20000 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
# and what they ask for
tail -n 20000 /var/log/nginx/access.log | awk '{print $7}' | cut -d'?' -f1 | sort | uniq -c | sort -rn | head

If a few addresses dominate, you have a simple flood. If the top address sends 1% of the traffic and the next thousand send about as much each, you have a distributed one, and per-address rules will not be enough.

How to protect yourself

Make requests cheap: cache

The most effective defence against a flood is to make most requests cost nothing. A full-page cache in front of PHP turns an expensive page into a file read. Even a one-second “microcache” in nginx collapses a flood on the home page into one PHP request per second:

fastcgi_cache_path /var/cache/nginx/micro levels=1:2 keys_zone=micro:10m max_size=200m inactive=60s;

server {
    location ~ \.php$ {
        fastcgi_cache micro;
        fastcgi_cache_key $scheme$request_method$host$request_uri;
        fastcgi_cache_valid 200 1s;
        fastcgi_cache_use_stale updating error timeout;
        fastcgi_cache_lock on;
        # never cache logged-in users
        fastcgi_cache_bypass $cookie_PHPSESSID;
        fastcgi_no_cache $cookie_PHPSESSID;
        # ... your usual fastcgi_pass / params
    }
}

fastcgi_cache_lock matters: it lets one request rebuild an expired entry while the others wait, instead of all of them hitting PHP. Adapt the bypass condition to your application’s session cookie.

Rate-limit per address

nginx’s limit_req caps the request rate per key, with a burst for normal browsing:

limit_req_zone $binary_remote_addr zone=dyn:20m rate=5r/s;

server {
    location ~ \.php$ {
        limit_req zone=dyn burst=20 nodelay;
        limit_req_status 429;
    }
    location = /search {
        limit_req zone=dyn burst=5;
    }
}

Apply it to dynamic routes, not to static assets: a single page view can load fifty images and scripts. On Apache, mod_evasive or mod_qos play the same role, less precisely.

Size the stack so overload stays contained

Set pm.max_children from memory (available RAM divided by a PHP process’s typical size), and make sure the sum across all pools stays below MySQL’s max_connections. When PHP is saturated you get 502s on the overloaded site; when the database is saturated, every site that shares it fails at once.

Know where the per-address limit stops

Against a flood from thousands of residential addresses, each sending a few requests a minute, no per-address threshold works without also catching real visitors. That is where a CDN or a WAF with bot management earns its price: it sees traffic across many sites, can challenge suspicious clients with JavaScript or a CAPTCHA, and absorbs the volume before it reaches you. Most offer an “under attack” mode for exactly this. Behind a CDN, configure mod_remoteip or nginx’s real_ip_header so your logs and limits see the visitor, not the CDN.

How Demerzel helps

Once the agent may read your web logs, Demerzel sees each address’s request counts, status codes and sampled paths, minute by minute.

• It separates noise from damage. The Security tab’s timeline stacks requests, the share coming from addresses a rule already flagged, the 5xx your sites returned and the machine’s load on one time axis. Demerzel opens an incident where an attack and a symptom meet — a real share of the traffic from flagged addresses within ten minutes of errors served to everyone else or a load above twice the vCPUs. A massive flood of flagged traffic opens one on its own. Each incident has a report: what happened, who was behind it, which sites suffered.
• It can slow a heavy address down. The rate-dynamic rule fires on more than 300 dynamic requests in two minutes from one address across all your sites (static assets do not count), and spread-vhosts on an address hitting more than eight of the server’s sites in ten minutes. With the warden enforcing them — after a replay on your own logs proved they would not have caught a real visitor — the first throttles the address’s new connections to your web ports, the second bans it.
• It refuses to cut off a big part of your traffic on its own. An address carrying more than 10% of the last hour’s successful requests is never blocked by a rule: an office behind one NAT, a partner’s integration. Demerzel raises an Action and you decide.
• It flags the configuration that turns an overload into an outage: PHP-FPM workers against database connections, the slow query log, real client IPs behind a proxy.

What it does not do: absorb a flood spread over thousands of addresses each staying under every threshold. Per-address rules cannot see that attack; a CDN or WAF can. When traffic arrives through Cloudflare, Demerzel says so in the address’s dossier and never blocks a Cloudflare address, since that would cut off every visitor behind it.