Slowloris and slow HTTP attacks: taking a server down one byte at a time

Most denial-of-service attacks try to send more than you can absorb. Slow HTTP attacks do the opposite: they send almost nothing, very slowly, and let your web server’s own patience do the damage. A single laptop on a home connection can make an unprotected Apache stop answering.

The names

• Slowloris — slow request headers. Published by Robert “RSnake” Hansen in 2009, along with the Perl script that gave the attack its name.
• Slow POST, also known as R.U.D.Y. (“R U Dead Yet?”) — slow request body.
• Slow read — slow response consumption. Popularised in 2012 by the slowhttptest tool, which implements all three.

They belong to the same family: low-and-slow, application-layer denial of service.

What the attacker wants

Your server can only work on a limited number of requests at once. Apache has a fixed pool of workers (Debian and Ubuntu ship MaxRequestWorkers 150 for both the event and prefork MPMs). nginx has worker_connections per worker process. PHP-FPM has pm.max_children. The attacker wants to occupy every one of those slots with a request that never finishes, so that real visitors queue and then time out. No crash, no exploit — the service is simply full.

The appeal for the attacker is the cost: a few hundred connections, each sending a few bytes every few seconds, is invisible on a bandwidth graph.

How it works

Slowloris: headers that never end

An HTTP request ends its headers with an empty line. Until that line arrives, the server cannot process the request, so it waits. Slowloris opens many connections and on each one sends a valid request line and a few headers, then every 10–15 seconds another header line:

GET / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0
X-a: 4127
X-a: 913

The blank line never comes. Each trickle resets the server’s read timer, so the connection stays alive as long as the attacker likes.

On a thread- or process-per-connection server such as Apache with the prefork or worker MPM, each of those connections holds a worker. The event MPM helps with idle keep-alive connections, which it parks without a worker, but a connection still in the middle of sending its request headers occupies a worker there too. nginx is event-driven and holds thousands of idle connections cheaply, which makes it much harder to exhaust this way — but not immune, since worker_connections is still a limit.

Slow POST: a body at walking pace

The attacker sends complete headers with a large Content-Length, say a 1 MB form submission, then sends the body one byte at a time. The server has accepted the request and handed it to a worker, which now waits for the rest of the body. Login forms and upload endpoints are ideal targets because they legitimately accept POST bodies.

Slow read: a client that won’t listen

Here the request is normal and quick. The attacker then reads the response extremely slowly and advertises a tiny TCP receive window, so the server can only push a few bytes at a time. The worker that generated the response stays busy writing it. Large responses — a big page, a file download — make it worse.

What it looks like on your server

The tricky part is that the access log stays quiet. A line is written when a request completes, and these requests do not complete. When they finally time out, Apache logs them with status 408 and often an empty request line ("-"). A sudden run of 408s is a strong hint.

The live view is more useful:

# Established connections to the web ports, per client address
ss -Htn state established '( sport = :443 or sport = :80 )' \
  | awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c | sort -rn | head

One address holding hundreds of connections is your culprit. On Apache, mod_status (/server-status) shows the scoreboard: a slow-headers attack fills it with R (“reading request”), a slow read with W (“sending reply”). When every slot shows one of those, Apache logs AH00161: server reached MaxRequestWorkers setting in its error log.

On the host itself, CPU and load stay low. That is the signature: the site is down, the machine is idle.

How to protect yourself

The defence is the same in every case: stop waiting forever, and stop letting one address take everything.

Apache

mod_reqtimeout, enabled by default on Debian and Ubuntu, is the specific answer to Slowloris and slow POST. It sets a deadline for headers and body, extended only while the client keeps a minimum rate:

# /etc/apache2/mods-available/reqtimeout.conf
<IfModule reqtimeout_module>
    # 20 s for headers, up to 40 s if the client sends at least 500 bytes/s
    RequestReadTimeout header=20-40,MinRate=500
    # 20 s for the body, extended at the same minimum rate
    RequestReadTimeout body=20,MinRate=500
</IfModule>

Then lower the general Timeout. It governs, among other things, how long Apache waits on a stalled read or write — which is what a slow read exploits. Debian’s apache2.conf sets it to 300 seconds (Apache’s own default is 60). Thirty seconds is plenty for a client; if PHP behind mod_proxy_fcgi legitimately runs longer, give the backend its own budget:

Timeout 30
KeepAliveTimeout 5
# For long PHP requests: ProxyTimeout defaults to Timeout, so set it explicitly
ProxyTimeout 300

Prefer the event MPM with PHP-FPM over prefork with mod_php: idle keep-alive connections then cost no worker.

nginx

nginx’s defaults are 60 seconds for each of these timeouts. Tighten them and cap connections per address:

http {
    client_header_timeout 10s;   # whole header must arrive within 10 s
    client_body_timeout   10s;   # max gap between two reads of the body
    send_timeout          10s;   # max gap between two writes to the client
    keepalive_timeout     15s;

    limit_conn_zone $binary_remote_addr zone=peraddr:10m;
    server {
        limit_conn peraddr 20;
    }
}

Note that client_body_timeout and send_timeout apply between two successive operations, not to the whole transfer: a client that sends one byte every 9 seconds still passes. The per-address connection limit is what makes that useless at scale.

Put a buffering proxy in front

nginx as a reverse proxy buffers the whole request before passing it upstream (proxy_request_buffering on is the default) and buffers the response before handing it back (proxy_buffering on). Slow clients then tie up cheap nginx connections instead of Apache or PHP workers. A CDN in front does the same thing at a larger scale.

Cap connections per address in the firewall

# Refuse an address's 51st simultaneous connection to the web ports
iptables -A INPUT -p tcp -m multiport --dports 80,443 --syn \
  -m connlimit --connlimit-above 50 --connlimit-mask 32 -j REJECT --reject-with tcp-reset

Behind a CDN or a load balancer, every visitor arrives from the proxy’s addresses, so a per-source limit would throttle everyone. Apply it only where clients connect directly.

How Demerzel helps

Honestly, Demerzel’s detection rules are built on access logs, and a slow HTTP attack writes almost nothing there until it times out. None of the Active Defense rules is designed to recognise Slowloris as it starts. What Demerzel does is make sure the outage does not go unnoticed and unexplained:

• It finds the outage. Demerzel reads requests per minute from your web logs and knows when it last heard from the agent. Minutes in which a normally busy server served nothing at all, while the agent stalled and the machine kept running, become an outage of the kind stopped answering — the shape held workers leave. (The agent checks your certificates through the local web server, so a web server with no free worker stalls it too.)
• It writes the post-mortem. Impact, timeline, what was banned just before, what the evidence cannot tell, and next steps: the exact ss command above to find the address holding connections, and how to free them. It is shown on the server page and sent to every channel subscribed to availability alerts, whatever their threshold.
• It checks the configuration attacks lean on. The agent reports PHP-FPM’s pm.max_children against MySQL’s max_connections, whether the slow query log is on, and whether your web server logs the real client behind a proxy — the settings that decide how far an overload spreads.

The next article in this series is about a twist on the same mechanism that we met on our own fleet: the outage caused not by the attacker, but by the ban.