When your own ban takes the site down

This one is not a new kind of attack. It is a failure mode of the defence, and we met it on our own fleet in October 2026. It is worth understanding because almost every server that runs fail2ban is exposed to it.

What happened

A shared web server running Apache on Debian, with the distribution’s defaults: MaxRequestWorkers 150, Timeout 300. A scanner started sweeping the sites it hosted and opened about 450 connections at once. fail2ban did its job: one of its jails matched, and it banned the address.

Then every site on the server stopped answering for about four minutes.

The machine was fine: it had not rebooted, and nothing on it had crashed. The sites simply did not respond, then came back on their own. Without something watching, it would have been an unexplained blip — a customer complaining about “the site was down for a few minutes this morning” and nobody able to say why.

Why the ban caused it

A firewall rule acts on packets, not on connections. When fail2ban bans 203.0.113.45, its action inserts a rule at the top of the input chain that rejects or drops every packet from that address — including the packets that belong to connections already established. That rule sits before the usual ct state established,related accept, which is what makes the ban effective.

Now look at the server’s side of those 450 connections:

1. About 150 of them each had an Apache worker attached, reading a request or writing a response. The rest were waiting in the listen backlog.
2. The ban dropped every packet the scanner sent afterwards: the rest of its requests, its ACKs, its FINs.
3. From the kernel’s point of view, these sockets are still perfectly valid ESTABLISHED connections whose peer has gone quiet. Nothing tells Apache to give up.
4. Each worker sits in its read or write until Apache’s Timeout fires — 300 seconds on Debian and Ubuntu.

For as long as that lasts, every worker belongs to an address that can no longer talk to the server. Real visitors complete the TCP handshake, then wait for a worker that never comes. The site is down, and the cause is the ban, not the attack.

The same happens with any tool that bans by firewall: iptables or nftables by hand, a cloud security group, UFW, a CrowdSec bouncer. And note that conntrack -D -s 203.0.113.45 does not help: it removes the firewall’s connection-tracking entries, not the server’s sockets.

What it looks like

Held connections are easy to count while they last:

ss -Htn state established '( sport = :443 or sport = :80 )' \
  | awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c | sort -rn | head

    448 [::ffff:203.0.113.45]
      3 [::ffff:198.51.100.20]
      1 [2001:db8:42::7]

One address with hundreds of connections, which also appears in fail2ban-client status <jail> as banned. Note the address format: Apache listens on an IPv6 socket that also accepts IPv4, so IPv4 clients appear as IPv4-mapped IPv6 addresses, ::ffff:a.b.c.d. That detail matters in a moment.

Apache’s error log shows AH00161: server reached MaxRequestWorkers setting and /server-status is full of workers in R or W state. The access log shows a hole: no requests at all for the duration, then a burst of 408s when the timeouts fire.

How to prevent it

1. Close the connections when you ban

Linux can destroy a socket on demand: ss -K (kill) closes every socket matching a filter. It needs root and a kernel built with CONFIG_INET_DIAG_DESTROY, which current Debian and Ubuntu kernels have. To free the server from a banned address:

ss -K dst 203.0.113.45 '( sport = :80 or sport = :443 )'
ss -K dst '[::ffff:203.0.113.45]' '( sport = :80 or sport = :443 )'

Two details make or break this command:

• The brackets. Written bare, ss reads ::ffff:203.0.113.45 as an address followed by a port, and refuses it. Brackets make it an address. Run both lines: the first catches IPv4 sockets, the second the mapped addresses on a dual-stack listener.
• The port filter. ss -K dst <ip> alone kills every connection to that address — including your own SSH session if, one day, the banned address is yours. Limiting it to the web ports keeps the fix from turning into a lockout.

Each worker blocked on one of those sockets gets an error immediately and goes back to the pool.

To do it at every ban, add a fail2ban action:

# /etc/fail2ban/action.d/close-web-connections.conf
[Definition]
actionban = ss -K dst "[<ip>]" '( sport = :80 or sport = :443 )' || true
            ss -K dst "[::ffff:<ip>]" '( sport = :80 or sport = :443 )' || true
actionunban =

and attach it next to the jail’s firewall action:

# /etc/fail2ban/jail.local
[apache-badbots]
enabled = true
action  = %(action_)s
          close-web-connections

The brackets work for plain IPv4 and IPv6 addresses too; the || true keeps fail2ban from logging an error when the mapped form does not apply, for example for an IPv6 ban.

2. Stop waiting five minutes

The ban is not the only thing that can leave a worker waiting on a silent client: a mobile visitor going into a tunnel does it too, and Slowloris does it on purpose (see the first article in this series). Apache’s Timeout is how long a worker waits, and 300 seconds is far too generous for talking to a browser:

# /etc/apache2/apache2.conf
Timeout 30
# Long PHP requests (exports, imports) keep their own budget:
# ProxyTimeout defaults to Timeout, so set it explicitly.
ProxyTimeout 300

ProxyTimeout governs the wait for the backend — PHP-FPM through mod_proxy_fcgi — so a 2-minute report still runs, while a vanished client releases its worker after 30 seconds. With both changes, a ban frees its connections immediately, and anything that slips through holds a worker for 30 seconds instead of 300.

Raising MaxRequestWorkers is not a fix. It moves the ceiling, and the next scanner opens more connections.

How Demerzel helps

This is how we found out — and why the outage did not stay a mystery.

• It detected the outage. Demerzel counts requests per minute from the server’s own web logs and knows when it last heard from the agent. A normally busy server serving nothing at all, an agent stalled (it checks certificates through the local web server, which had no worker left), and a machine that had not rebooted: that is an outage of the kind stopped answering.
• It wrote the post-mortem. Impact (how long visitors were cut off, roughly how many requests went unserved), a timeline including the fail2ban ban just before the cut — read from fail2ban’s log once the agent is allowed to — what the evidence cannot tell, and next steps: the ss command to find the address holding connections, the ss -K command to free them, and the Timeout / ProxyTimeout change. It appeared on the server page and in the inboxes of every channel subscribed to availability alerts, nobody having asked.
• Its own bans close what they leave open. Demerzel’s warden, the optional component that applies Active Defense bans in its own nftables table, closes the web connections a banned address holds at the moment it bans it, so a warden ban cannot freeze your workers this way. Like every warden action, it stays within the root-owned policy: allowlisted addresses and operators logged in over SSH are never banned in the first place.

A ban that takes the site down is a defence working against you. Closing the connections takes one line; finding out you needed it took a post-mortem.