DDoS Testing for NGINX and Load Balancers: Slots, Limits, and Upstream Health

The tier that fails first in many web stacks is the one nobody puts in the test plan.

A CDN gets tested because it is the famous layer. The application gets tested because it is the expensive one. The reverse proxy or load balancer in between, usually nginx, sometimes HAProxy, often a cloud balancer in front of an nginx fleet, gets assumed. It was put there to remove a bottleneck, so it is rarely suspected of being one.

Under a flood it often is. Its worker slots, its keepalive policy, its rate-limit directives and its view of upstream health all have limits and defaults that behave differently under burst than they read in the config file. This post is about finding those limits on purpose.

What DDoS testing for NGINX and load balancers is

DDoS testing for NGINX and load balancers is a controlled test of the proxy tier itself: how many connections it can hold before new users are refused, whether limit_req and limit_conn do under burst what they appear to do on paper, and whether its upstream health logic keeps serving or converts a partial slowdown into a total outage.

What a balancer is and how the algorithms spread load is covered in the load balancer definition; how a DDoS testing engagement is scoped and run on any target is in the complete guide. Neither is repeated here, and neither is the pod-and-autoscaler layer around nginx when it runs as a cluster ingress, which belongs to Kubernetes DDoS testing. What follows is nginx's own arithmetic, nginx's own defaults, and the ways the proxy tier fails while the backends behind it look fine.

nginx control	What the config suggests	What happens under burst	What a test measures
`worker_connections`	a connection ceiling per worker	every proxied request spends two slots	the client count at which accept stalls
`keepalive_timeout`	a courtesy to returning clients	idle sockets are reclaimed early, half-read requests are not	which kind of connection is holding the slots
`limit_req`	N requests per second per key	one request per 1/N second, burst queued or refused	observed rejections vs configured rate
`limit_conn`	N connections per key	counts only connections with a request in progress	whether idle and slow clients are counted at all
`max_fails` / `fail_timeout`	health checking	live traffic failing marks servers out	whether the pool empties under load
upstream `keepalive`	connection reuse to backends	off by default: a new TCP connection per request	new upstream connections per second

The slot budget is smaller than it looks

nginx is event-driven, so a connection costs a slot in a worker's table rather than a whole thread. That is why it shrugs off loads that sink a prefork server. It is also why the number that matters is easy to misread.

Every proxied request costs two slots

worker_connections counts all connections a worker holds, not only clients. The documentation says so directly: the number includes connections to proxied servers.

So a proxied request in flight costs one client connection plus one upstream connection. With worker_processes 4 and worker_connections 1024, the ceiling is 4,096 connections, which is 2,048 proxied requests in flight, not 4,096. The compiled default for worker_connections is 512; packaged configs commonly ship 768 or 1024. Many production proxies run on whichever number the package installed.

File descriptors come before either number

Each connection is a file descriptor. If the process limit (ulimit -n, or worker_rlimit_nofile in the config) is lower than worker_connections, the descriptor limit is the real ceiling and the log shows Too many open files instead.

Cached file handles, log files and upstream sockets draw from the same budget. A test that reaches the configured ceiling without seeing the descriptor error has confirmed something useful; a test that hits the descriptor error first has found the actual number.

What the log says when it runs out

When a worker runs out of slots, nginx logs worker_connections are not enough and stops accepting until one frees. New connections then wait in the kernel's accept queue, whose length nginx sets with the backlog parameter on listen (511 by default on Linux, capped by net.core.somaxconn). When that fills too, the kernel drops or resets new handshakes.

None of this shows on a backend dashboard. The backends see fewer requests, not more.

Diagram of an nginx reverse proxy with all 4,096 connection slots held by slow clients sending one byte of request body every 50 seconds, requests nginx is still reading and cannot reclaim. A real user's new connection is not accepted, and the error log reads worker_connections are not enough. Behind the proxy, three backend servers show zero requests in flight and idle CPU, because with request buffering on nginx reads the whole body before proxying. The proxy is the layer of first failure while every backend dashboard reads healthy.

Keepalive and slow clients: who is holding the slot

A slot is held for as long as a connection lives, whether or not anything useful is happening on it. That makes the timeouts, more than the connection count, decide how many clients the proxy can actually serve.

Idle keepalive is reclaimable, mid-request is not

keepalive_timeout defaults to 75 seconds, and a client that finishes a request and goes quiet holds its slot for that long. nginx has a safety valve here that many operators do not know about. When a worker runs short of free slots, it closes idle keepalive connections early to make room and logs worker_connections are not enough, reusing connections.

That valve only covers connections nginx considers reusable: idle between requests, or open with nothing sent yet. A connection with a request partly read is not reusable. So an attacker who wants to keep a slot has to keep a request open, and the test question becomes which kind of holder is filling the table. If it is idle keepalive, nginx is already reclaiming it. If it is half-read requests, only the timeouts will.

Header and body timeouts are not the same kind of timeout

nginx's client_header_timeout (60 s by default) bounds how long the client has to deliver the request header, which is why a header-dribbling Slowloris is far less effective against it than against a thread-per-connection server.

client_body_timeout is different. It applies only between two successive reads, not to the whole body, so a client that sends one byte every 50 seconds under a 60-second timeout can hold the connection indefinitely. That is the RUDY shape, and it lands on nginx specifically because of that per-read definition. send_timeout works the same way on the response side, which is the opening for a slow read.

Request buffering moves the stall onto the proxy

With proxy_request_buffering on (the default), nginx reads the full request body before it opens an upstream connection. That is protective: a slow uploader never ties up a backend worker.

It also means the stall now lives entirely on the proxy, which is the point of the diagram above. The backends are protected, and the proxy absorbs every slow body on its own slot budget. Turning buffering off for large uploads or streaming moves the problem back onto the upstream and doubles its slot cost on the proxy for the whole transfer.

Holding an attacker costs you the same slot

A tarpit is attractive in theory: keep the hostile connection open and unproductive. On an nginx proxy it is self-defeating, because the held connection occupies exactly the same worker_connections slot a real user needs. The nginx-native opposite is return 444, which closes the connection with no response at all and frees the slot at once.

Step-to-cap model chart of nginx connection slots in use. 200 new client connections arrive per second. From fifteen seconds in, the time each client holds its connection grows geometrically from 50 milliseconds toward the 60 second body timeout, and by Little's law the slots in use climb from about ten to the cap of 4,096, four workers times 1,024 worker_connections, where the line goes flat. Model, not a measurement.

The chart is Little's law and nothing else: slots in use equal the arrival rate times the time each connection is held. At 200 new connections a second, a 50 ms hold uses about ten slots. A hold time of roughly 20 seconds uses all 4,096. The arrival rate never changed.

`limit_req` under burst, not on paper

limit_req is the directive most people point to when asked how the proxy is rate limited. It is a leaky bucket, and the way it counts is the first surprise.

10r/s means one request every 100 ms

nginx tracks the rate at millisecond granularity. rate=10r/s is enforced as one request per 100 ms. With no burst, a second request arriving 40 ms after the first is rejected, even if the client sends only two requests that second.

A single browser page load can open several requests inside 100 ms. A limit written as "ten per second, generous" can therefore refuse ordinary users on the first page, and that is a finding a test should surface before an incident does.

`burst` queues, `nodelay` spends, `delay` splits

burst=5 lets five excess requests wait instead of failing. They are released at the configured rate, so the sixth request in a burst waits 500 ms. While it waits, it holds its client connection and its slot.

nodelay serves the burst immediately and only uses the queue as an accounting allowance, which refills at the configured rate. delay=N (since nginx 1.15.7) serves the first N without delay and queues the rest. The three settings produce very different latency and rejection curves under the same attack, and the config file does not tell you which one a real burst will hit.

Diagram of an nginx limit_req rule with rate 10 requests per second applied to six requests arriving within 100 milliseconds. With no burst, one request is served and five are rejected with 503, the default limit_req_status. With burst=5, all six are served but released one per 100 milliseconds, so the sixth waits 500 milliseconds holding its connection. With burst=5 nodelay, all six are served at once and the five burst slots refill at one per 100 milliseconds, so a quick repeat burst is mostly refused.

The rejection is a 503 unless you changed it

Both limit_req_status and limit_conn_status default to 503. A rate-limited client and a broken backend therefore return the same status code, which makes the load-shedding invisible in most dashboards and indistinguishable from an outage in synthetic monitoring. Setting 429 is one line. Whether it was set is a test result.

The key is only as good as the client address

Most examples key on $binary_remote_addr. Behind a CDN or an L4 balancer that is the upstream hop, not the client, so every user shares a handful of addresses and one bucket. The realip module fixes this with set_real_ip_from and real_ip_header, and it is only safe when set_real_ip_from lists the actual front-end ranges, otherwise any client can set the header. Why per-IP keys fail for other reasons belongs to DDoS testing for APIs.

The zone can fill, and then it evicts

The shared-memory zone holds one state per key. The documentation puts it at about 16,000 64-byte states per megabyte. When the zone is full, nginx evicts the least recently used state; if it still cannot create one, the request fails with the limit's error status.

A flood from many distinct addresses can therefore churn a small zone until the limiter forgets the keys it was meant to be throttling. A test that uses more sources than the zone can hold shows whether that happens.

The limit is per node, not per fleet

The zone is shared across one server's workers and nothing further. Four nginx nodes behind a cloud balancer, each with rate=10r/s, give a single client an effective 40 r/s if the balancer spreads its connections. The limit that was agreed in a design review is not the limit in production unless the balancer pins clients to nodes.

`limit_conn` counts less than you think

limit_conn looks like the direct answer to slow and idle connections. The documentation says otherwise.

A connection is counted only if it has a request being processed and the whole request header has already been read. An idle keepalive connection is not counted. A client still dribbling its header is not counted. The two connection shapes that most need a per-client ceiling are the two the directive does not see.

Under HTTP/2 the opposite happens: each concurrent stream is counted as a separate connection. A limit_conn perip 10 that was sized for HTTP/1.1 browsers can refuse an ordinary HTTP/2 client that multiplexes more than ten requests, while doing nothing about a thousand idle HTTP/1.1 sockets from the same range.

The test for this is short. Open idle and slow connections from one source until they exceed the configured limit_conn, and check whether any are refused. Then repeat with multiplexed HTTP/2 streams from a well-behaved client, and check that it is not.

Health checks that empty the pool

How health checks can cascade across a pool is covered generically in the load balancer definition. What is specific to open-source nginx is that it has no active checks at all: the health_check directive is an NGINX Plus feature. The only signal is real traffic failing.

The defaults are aggressive

max_fails=1 and fail_timeout=10s are the defaults. One failed attempt, as defined by proxy_next_upstream, marks a server unavailable for ten seconds. The default for proxy_next_upstream is error timeout, so a single timed-out request is enough.

That same directive then retries the request on the next server. proxy_next_upstream_tries defaults to 0, which means no limit, so one slow client request can walk the entire pool. Non-idempotent methods such as POST are not retried by default (since 1.9.13), which spares writes and leaves every GET free to walk.

How the pool empties

Under load, one backend crosses its timeout, is marked out, and its share plus its retries land on the others. They slow, time out, and are marked out within the same ten-second window. When every server in the group is unavailable, nginx logs no live upstreams while connecting to upstream and answers with an immediate 502.

The backends may already be recovering. The proxy is still refusing to send them traffic. The outage is now enforced by the component meant to route around one.

A single-server upstream group behaves differently: with only one server, nginx ignores max_fails and never marks it unavailable. Tests on a staging stack with one backend will never reproduce the cascade that production's pool of three produces.

Four-stage diagram of nginx passive health checks under load. Stage one, backend 1 times out once, which meets max_fails=1, so it is marked out for fail_timeout of 10 seconds and the same request is retried on backend 2 by proxy_next_upstream. Stage two, backend 2 inherits its own share, backend 1's share and the retries, times out once and is marked out. Stage three, backend 3 takes all traffic and times out within the same 10 second window. Stage four, nginx logs no live upstreams while connecting and returns instant 502s to everyone even as backends recover. Defaults shown: max_fails=1, fail_timeout=10s, proxy_next_upstream error timeout, proxy_next_upstream_tries 0. Active checks are an NGINX Plus feature.

What to change, and what to prove

Raising max_fails, capping proxy_next_upstream_tries at two, setting proxy_next_upstream_timeout, and making proxy_read_timeout realistic rather than the 60-second default all reduce the risk. None of them prove anything until the test injects latency into one backend and watches the pool state through the whole event, including the ten seconds after the latency is removed.

The side facing the backends

The proxy's upstream connections are the half of the tier that rarely makes it into a diagram.

No upstream keepalive unless you ask for it

By default nginx speaks HTTP/1.0 to upstreams and closes the connection after each response. Every proxied request opens a new TCP connection to a backend. Reuse needs three things together: keepalive N in the upstream block, proxy_http_version 1.1, and proxy_set_header Connection "". Any one missing and the pool is not used.

The ephemeral port ceiling

Each new upstream connection takes a local port. The Linux default ip_local_port_range is 32768 to 60999, which is 28,232 ports per backend address and port. When the proxy is the side that closes, each port sits in TIME_WAIT for 60 seconds. 28,232 divided by 60 is about 470 new connections per second to one backend, sustained.

Past that, nginx logs connect() failed (99: Cannot assign requested address) while connecting to upstream. It is an outage caused by a flood the backends could have served, and upstream keepalive removes most of it.

Two conntrack entries per request

If the proxy host runs a stateful firewall, each proxied request is two tracked flows, one in and one out. The proxy's conntrack table can therefore fill at half the client connection rate a backend's table would.

When the balancer in front is the bottleneck

Many nginx tiers sit behind a cloud or hardware L4 balancer. That adds its own ceilings and two nginx-specific interactions worth testing.

The outer balancer health-checks nginx

When nginx is saturated, the outer balancer's health checks to it slow or fail. Nodes are pulled from rotation, the survivors take more, and the same pool-emptying pattern repeats one layer out. A health-check endpoint served by nginx itself (a location returning 200 from memory) will report healthy while every proxied request is queued, so decide which of those two behaviours you want before the test tells you which one you have.

Client addresses need the PROXY protocol

An L4 balancer that does not preserve the source address hands nginx its own address on every connection. Every limit_req and limit_conn keyed on the client then keys on the balancer. listen ... proxy_protocol and set_real_ip_from with real_ip_header proxy_protocol restore it. A test run from many sources is the quickest way to find out whether the per-client limits are actually per client.

Other proxies, same questions

HAProxy exposes the same pressures through different knobs: a global and per-frontend maxconn, a per-server maxconn that queues requests inside HAProxy with a timeout queue, and stick tables for per-key rates. The questions are identical. How many connections are held, what is counted, what is queued and for how long, and when does a failing backend get pulled.

Running the test

Proxy-tier testing is unusually easy to aim at production by accident, because the proxy is shared by everything behind it. The containment mechanics (scoped targets, rate ceilings, abort criteria) are in DDoS testing without disrupting production. Four things are particular to this tier:

Measure connections, not requests per second. Drive idle and slow connections at a known rate and record the count at which a fresh client's handshake stalls. That number should match worker_processes × worker_connections, or the descriptor limit, or the accept backlog. Knowing which one it matched is the result.
Read stub_status throughout. Active, Reading, Writing and Waiting tell you who holds the slots. Waiting is idle keepalive nginx can reclaim. Reading is clients still sending headers, capped by client_header_timeout. Once the header is in, nginx counts the connection under Writing, so slow-body clients sit there next to real proxied work. A high Writing count while the backends sit idle is the slow-body signature.
Compare observed limits to configured ones. Plot rejections against the offered rate from a single key, then from many keys, then through the outer balancer. Differences are findings.
Inject latency, not load, for the health-check test. Slow one backend deliberately and record the upstream pool state every second. If the pool empties, record how long the 502s outlast the injected fault.

FAQ

What does DDoS testing for NGINX actually test?

It tests the proxy tier on its own terms: the connection count at which nginx stops accepting, whether limit_req and limit_conn behave under burst the way the configuration implies, how passive health checks react when backends slow, and whether the upstream side runs out of ports or connections. The backends can be healthy through all of it, which is why the tier needs its own test.

Why is nginx often the layer that fails first?

Because every proxied request spends two connection slots, slow uploads hold slots without doing work, and request buffering deliberately keeps those stalls on the proxy rather than on the backends. The proxy runs out of slots while the backends are quiet.

Does limit_conn stop slow or idle connections?

Not on its own. nginx counts a connection toward limit_conn only once it has a request in progress and its whole header has been read. Idle keepalive connections and clients still sending headers are not counted, so the controls for those shapes are the header and body timeouts, nginx's own reclaiming of idle keepalive, and closing hostile connections early with return 444.

What does "no live upstreams" mean during a load test?

It means nginx has marked every server in the upstream group unavailable through its passive health checks, usually because timeouts under load met max_fails on each in turn while proxy_next_upstream retried requests across them. nginx then returns 502 immediately until fail_timeout expires. It is a proxy decision, and it can outlast the backend problem that caused it.

Is a cloud load balancer in front of nginx enough protection?

It adds capacity and its own ceilings, and it changes what nginx sees. Without the PROXY protocol or a trusted real-IP header, every per-client limit in nginx keys on the balancer's address. And the outer balancer's health checks can pull saturated nginx nodes out of rotation, repeating the pool-emptying pattern one layer out.

Nothing here is exotic

Every failure in this post comes from a default, a counting rule or a timeout definition that is written down in the nginx documentation. worker_connections includes upstreams. limit_conn ignores idle sockets. max_fails is 1. Upstream keepalive is off. None of it is hidden, and almost none of it is read until the proxy is the thing that fell over.

The proxy tier is a pile of small, documented numbers. An attacker only has to find the smallest one, and the test is the only reliable way to know in advance which one that is.