Handling Server Resource Exhaustion: Emergency Response Strategies for High Traffic Sites

When “sudden success” looks like failure

A product launch goes viral. A sale gets syndicated on deal sites. A news article mentions your brand. Or a botnet decides you look interesting. In minutes, requests flood in, and what felt like comfortable headroom yesterday vanishes today. CPU spikes. Database connections saturate. Queues back up. Users see timeouts. Alert fatigue sets in.

This is server resource exhaustion: the moment demand for compute, memory, I/O, or critical connection pools exceeds the capacity your system can deliver. It’s both a technical and an operational challenge, and how you respond in the first 15 minutes often determines whether you stabilize quickly or spiral into a full outage.

This guide covers practical, battle-tested emergency response strategies for high traffic surges—what to do right now, what to turn on next, and what to build to make the next surge routine rather than rattling.

Recognize resource exhaustion before it hits the wall

Early detection gives you room to maneuver. Common symptoms include:

• CPU pegged near 100%, high load average, and rising latency.
• Memory pressure, frequent GC, swapping, or OOM kills.
• Database connection pool saturation; spikes in lock wait time, deadlocks, or slow queries.
• Thread pools maxed out (web servers, workers, caches).
• File descriptor exhaustion; too many open files.
• Disk I/O wait increasing; queue depth spikes; log volume explosions.
• Network bottlenecks: high packet drops, syn backlog overflow, or LB connection errors.
• Queue latency and length rising faster than worker throughput.
• Cache miss rate climbing unexpectedly (invalidations, hot content not caching).
• Error budgets draining: 5xxs, timeouts, and retries compounding load.

Build dashboards keyed to the USE and RED methods:

• USE: Utilization, Saturation, Errors for each resource.
• RED: Rate, Errors, Duration for each service/endpoint.

Set alerts not only on error rates, but on saturation leading indicators (e.g., 80% of database connections used, 70% cache miss rate, mean queue latency > X).

The first 15 minutes: stabilize and buy time

Think in layers. Your goal is to reduce work per request, protect critical paths, and shed non-essential load. Move fast, document actions, and prefer reversible changes.

1. Put someone in charge and communicate

• Assign an incident commander and scribe.
• Declare an incident in your chat tool; page on-call.
• Post a status page note: “Elevated errors; applying mitigations.”
• Pause deploys.

2. Reduce incoming load at the edge

• Enable CDN “under attack” mode or basic bot mitigation.
• Turn on temporary rate limiting per IP or per session on costly endpoints (checkout remains open; bulk search throttled).
• Block obviously abusive patterns (scrapers hammering search, login sprays).

Example (Nginx rate limit burst with token bucket):

http {
  limit_req_zone $binary_remote_addr zone=api_rate:10m rate=5r/s;

  server {
    location /api/search {
      limit_req zone=api_rate burst=20 nodelay;
      # upstream config...
    }
  }
}

3. Serve cheaper responses

• Force static caching for anon traffic; increase CDN TTLs on popular pages.
• Strip personalization for anon users to improve cache hit rate.
• Temporarily downgrade dynamic widgets, carousels, and recommendations.

4. Protect the database

• Cap per-service connection counts and turn on statement timeouts.
• Route read-heavy traffic to replicas; avoid promoting heavy writes.
• Kill runaway queries and pause non-essential cron jobs or ETL.

5. Control queues and backpressure

• Pause low-priority producers; keep SLO-critical jobs flowing.
• Increase worker concurrency if CPU headroom exists; otherwise limit concurrency to avoid thrashing.
• Drop or defer non-critical work (email digests, analytics enrichment).

6. Autoscale with guardrails

• Add capacity if you can, but avoid runaway autoscaling that amplifies costs or causes cold starts.
• Prefer step scaling with warm pools or pre-warmed instances.

7. Instrument and verify

• Watch success rate and p95/p99 latency; ensure mitigations help.
• Roll back any change that makes error rates worse.

This framework buys time without sacrificing the business-critical paths (e.g., checkout, login, content delivery) and helps you avoid turning a busy system into a broken one.

Traffic control: rate limit, shape, and filter

When surges come, not all traffic is equal. You need quick controls to prioritize humans and revenue paths.

Fast edge strategies

• CDN protections: Turn on bot fight modes, WAF managed rules, and bot scoring. Block or challenge high-risk requests with CAPTCHA for expensive endpoints.
• IP reputation: Temporarily block noisy ASNs or cloud provider ranges if clear bot patterns appear. Ensure allowlists for partners.
• Per-path policy: Apply stricter limits to endpoints with heavy DB reads or joins, e.g., faceted search, report export, or admin analytics.

Cloudflare example rules:

• Rate limit /login and /search at 10-20 r/min per IP.
• Challenge user-agents with missing Accept headers hammering APIs.
• Cache everything for GET on /product/* with 2–5 minute TTL if safe.

HAProxy stick-table for request rate limiting:

frontend fe_http
  bind *:80
  stick-table type ip size 100k expire 10m store gpc0,conn_rate(60s),http_req_rate(10s)
  http-request track-sc0 src
  acl too_many req_rate(sc0) gt 50
  http-request deny if too_many

Token buckets and fairness

• Use token bucket rate limiting to smooth bursts while allowing brief spikes.
• Implement per-user or per-session limits to avoid penalizing NATed office IPs.
• Be careful with global limits that punish legitimate surges; shape traffic rather than block when possible.

Serve fewer bytes

• Compress responses (Brotli/Gzip) and optimize payloads.
• Reduce response size by omitting non-essential fields for high-traffic endpoints during the surge.

Serve cheaper pages: cache first, render later

Caching is the fastest way to reduce origin load.

Quick wins

• CDN “cache everything” rules for anonymous GET pages with short TTL (30–180 seconds).
• Surrogate keys/tags: Purge by tag to update specific content without nuking the entire cache.
• Edge-side includes (ESI): Compose dynamic bits with a cached shell.

Nginx microcaching example:

location / {
  proxy_cache my_cache;
  proxy_cache_valid 200 302 1m;
  proxy_cache_valid 404 10s;
  add_header X-Cache-Status $upstream_cache_status;
}

De-personalize temporarily

• Fallback to generic “bestsellers” instead of per-user recommendations.
• Defer personalized calls to asynchronous fetches with timeouts; show defaults if personalization is slow.

Cache database results

• Hot keys: Increase TTL for hot cache keys (product details, category pages).
• Avoid cache stampede: Add jitter to TTLs and use single-flight locks to prevent thundering herds on cache misses.

Protect the database: pool, throttle, and shed work

Databases are often the first to saturate under load. Your emergency playbook should focus on keeping connections stable, queries bounded, and writes safe.

Levers you can pull fast

• Use a connection pooler (e.g., PgBouncer) in transaction pooling mode to multiply effective concurrency.
• Set timeouts aggressively:
  • PostgreSQL: statement_timeout (e.g., 3–10s), lock_timeout (500ms–2s), idle_in_transaction_session_timeout.
  • MySQL: max_execution_time or per-session timeouts.
• Cap per-app connection limits. Avoid each pod creating dozens of idle connections.

PgBouncer example:

[databases]
app = host=db-primary dbname=app_db pool_size=200

[pgbouncer]
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 50
query_timeout = 10

Shed and prioritize

• Read replicas: Route read-heavy features to replicas. Mark non-critical features read-mostly.
• Kill runaway queries: Use pg_stat_activity or performance_schema to find queries > 10s and terminate.
• Temporarily disable heavy jobs: nightly reports, backfills, large exports.
• Reduce per-query memory: Lower work_mem to avoid memory bloat per connection.

Safeguard write paths

• If write amplification is high (cascades, triggers), consider a temporary read-only mode for non-critical services while leaving checkout or order placement enabled.
• Add rate limiting to endpoints generating heavy writes (bulk imports, admin edits).
• For OLTP databases, avoid emergency indexing during peak; instead, defer to after stabilization unless the index is trivial and risk is low.

Queues and background jobs: backpressure beats backlog

Queues give you elasticity, but during surges they can silently absorb load until consumers drown.

Emergency actions

• Pause non-critical producers: analytics, event forwarding, low-priority webhooks.
• Prioritize queues: SLO-critical (payments, order confirmation) get dedicated workers and higher concurrency.
• Set max in-flight per worker; better to process steadily than to thrash and time out.
• Enable dead-letter queues with short retry budgets to avoid retry storms.

Control retry storms

• Exponential backoff with jitter. Cap retries. Collapse duplicate jobs at enqueue time when possible.
• If a downstream service is struggling, flip a circuit breaker to fail fast and reduce job generation.

Scale, but don’t chase the wave blindly

Autoscaling is powerful but can be a double-edged sword if it reacts slowly or overshoots.

Make autoscaling helpful

• Step scaling: Increase by larger steps early (e.g., +30%), then smaller increments as latency recovers.
• Pre-warm: Keep a warm pool of instances to reduce cold start impact.
• Scale on saturation, not just CPU: queue depth, p95 latency, and connection saturation are better signals.
• Put upper bounds: Prevent runaway scaling that causes cascading DB load.

Scale the right layer

• If DB is the bottleneck, scaling web servers won’t help. Scale read replicas or turn on read-through caches.
• For stateful services, prefer vertical scaling temporarily if horizontal scaling requires data movement.

Degrade gracefully: brownouts, feature flags, and SLOs

Graceful degradation turns a binary outage into a partial but usable experience.

• Feature flags: Toggle off expensive features (live chat, recommendations, auto-suggestions).
• Brownouts: Reduce the rate of expensive operations (search suggestions 1/3 requests, prefetch disabled).
• Static fallbacks: Serve cached product pages with “live inventory might be delayed” banners.
• 503 with Retry-After: Signal clients and crawlers to back off.
• Limit concurrency: Per-endpoint concurrency caps can stabilize latency at the cost of queued requests.

Example: In API gateways, apply per-route concurrency limits and queue length caps; fail fast when the queue exceeds N to prevent tail latencies exploding.

Systems-level triage: don’t ignore the kernel

Sometimes the bottleneck is below your app.

File descriptors and process limits

• Increase open file limits for web servers and DB clients:
  • ulimit -n 65536 (ensure persistent via systemd LimitNOFILE).
• Check for fd leaks with lsof and monitor per-process open FD count.

Network and sockets

• Tune SYN backlog and reuse:
  • net.ipv4.tcp_synack_retries, net.core.somaxconn, net.ipv4.tcp_tw_reuse (modern kernels), backlog in listen().
• Raise ephemeral port range if ephemeral exhaustion occurs:
  • net.ipv4.ip_local_port_range = 1024 65000
• Ensure NAT gateways/LBs aren’t the bottleneck (conn limits, PPS caps).

Disk and logs

• Logging volume spikes can throttle disks. Temporarily reduce log verbosity.
• Move access logs to async or buffer them more aggressively. Rotate logs if partitions fill.

Garbage collection and runtimes

• JVM/CLR/Node: Tune GC pauses by reducing heap pressure (lower concurrency, smaller in-flight).
• Limit per-process memory to prevent OOM cascades; prefer cgroups and memory limits with oom_score_adj tuned so critical processes survive.

Differentiate surge from attack

Surges can be organic or malicious; your response differs.

• Patterns of requests:
  • Organic: spikes on product pages, checkout, marketing landing pages, diverse referrers.
  • Attack: high RPS on login/auth, random URLs, unusual user-agents, no referrer, identical request signatures.
• Geography and ASN:
  • Organic: mixes of expected geos and ISPs.
  • Attack: concentrated in specific data centers or suspicious ASNs.
• Behavior:
  • Organic: resource fetches (CSS/JS) proportionate to HTML; sessions persist.
  • Attack: abnormal header sets, missing assets, high error tolerance.

If attack suspected:

• Tighten WAF rules and enable challenges on suspicious endpoints.
• Lower rate limits, raise penalties for noisy actors.
• Coordinate with your CDN or DDoS provider; they have faster, broader levers.

Communication: keep users and teams informed

During an incident:

• Internal comms:
  • Single channel for updates.
  • Roles: incident commander, ops lead, app lead, comms owner.
  • Time-boxed updates (every 10–15 minutes).
• External comms:
  • Status page updates with plain language.
  • Social or in-app banner: “Increased traffic; we’ve limited certain features to stay available.”
  • Post-incident summary with actions taken.

Clarity reduces duplicate support tickets and builds trust.

After the storm: harden for the next one

Emergency response buys you time. Permanent fixes ensure you won’t need the same heroics again.

Capacity planning and SLOs

• Set SLOs per critical user journey (e.g., 99% of checkouts < 3s).
• Model traffic multipliers (x2, x5, x10). Identify the first bottleneck at each multiple.
• Adopt a set capacity buffer (20–40%) or ensure your autoscaling plus warm pools can absorb a sudden x2.

Load testing that reflects reality

• Reproduce traffic shape and mix: reads vs writes, authenticated vs anon.
• Include cache-cold scenarios and cache invalidation storms.
• Test upstream integrations and downstream webhooks; many surges fail at third-party limits.

Architecture patterns for resilience

• Edge caching and full-page caching for anon pages.
• Request coalescing to stop stampedes.
• Bulkheads: isolate services so one failure doesn’t propagate.
• Circuit breakers with graceful fallback content.
• Queue backpressure with clear priorities and drop policies.
• Idempotency keys for write endpoints to handle retries safely.
• Multi-tier caching: CDN, application cache, database cache.

Database improvements

• Connection poolers by default; transaction pooling for chatty apps.
• Slow query remediation and appropriate indexes.
• Read replicas and read routing with health-aware clients.
• Partitioning or sharding plans if single-node scaling hits limits.

Observability you can act on

• Golden signals dashboards per service.
• Auto-created runbooks linked to alerts.
• Logging budgets and sampling to prevent storm-induced log amplification.

Operational readiness

• Runbook drills: “Turn on surge mode” should be a documented, one-click or scripted action.
• Feature flags for degradation; test them quarterly.
• Preconfigured CDN/WAF rules you can toggle instantly.
• On-call rotations with shadowing and simulations for load spikes.

Practical scenarios and what to do

E-commerce flash sale

• Before: Warm caches for top products, raise CDN TTLs, pre-scale read replicas, enable rate limiting on search autocomplete.
• During: Freeze personalized recommendations, cache category pages, limit export/report endpoints, prioritize checkout APIs.
• After: Analyze slow queries triggered by faceted search; add indexes; create a “sale mode” flag.

News site linked by a major platform

• Before: “Cache everything” rule for article pages with 120s TTL and tag-based purge.
• During: Strip non-essential JS, defer comments and personalization, rely on edge assemble.
• After: Build origin shield and fine-grained purge for breaking updates.

SaaS bulk import spike

• Before: Enforce per-account concurrency caps and background job quotas.
• During: Queue imports with fair scheduling; elevate UI responsiveness by deprioritizing imports; communicate ETAs.
• After: Implement adaptive throttling and per-tenant budgets.

Command quick-reference for triage

Use carefully and always coordinate with your team.

• CPU, memory, load:
  • top, htop, vmstat 1, dstat, sar -q
• Disk I/O:
  • iostat -xz 1, pidstat -d 1, iotop
• Network:
  • ss -s, ss -ant state established, netstat -s, ip -s link
• File descriptors:
  • lsof -p | wc -l, cat /proc/sys/fs/file-nr
• Database (PostgreSQL):
  • SELECT pid, state, wait_event, query, now() - query_start AS age FROM pg_stat_activity ORDER BY age DESC;
  • SELECT bl.pid, a.query FROM pg_locks bl JOIN pg_stat_activity a ON a.pid=bl.pid WHERE NOT bl.granted;
• Kill long-running query (PostgreSQL):
  • SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE now() - query_start > interval '10 seconds';

A 60–15–60 checklist

Keep a lightweight checklist handy to run consistently during incidents.

First 60 seconds

• Declare incident; assign roles; pause deploys.
• Toggle CDN/WAF surge mode and basic rate limits.
• Post status update.

First 15 minutes

• Increase cache TTLs; enable microcaching.
• Reduce personalization; disable heavy features via flags.
• Cap DB connections; set statement timeouts; kill runaway queries.
• Pause non-critical jobs; prioritize critical queues.
• Add capacity with guardrails; observe latency and error rate.

Next 60 minutes

• Fine-tune rate limits per endpoint; add IP/ASN blocks if needed.
• Right-size autoscaling; pre-warm if sustained traffic continues.
• Restore features gradually while monitoring SLOs.
• Document actions and timings; prepare post-incident review.

Build your “surge mode” now

The best emergency response is one you can enable with a switch. A minimal surge mode might include:

• CDN rules: cache-all for anon GETs; bot challenges on expensive endpoints.
• Nginx/HAProxy configs: per-path rate limits and concurrency caps.
• Feature flags: toggle off recommendations, export/reporting, heavy analytics.
• DB guardrails: global statement timeout, PgBouncer transaction pooling, read routing.
• Queue policies: priority queues; producer throttles; retry backoff defaults.
• Observability: a single “surge” dashboard with key signals and runbook links.

Bundle these into a tested playbook so the next traffic spike is a validation of your engineering, not a gamble.

Final thought

High traffic doesn’t have to mean high anxiety. With clear priorities—protect the database, serve cheaper pages, shed non-essential load—and a practiced set of toggles, your team can turn “sudden success” into sustained availability. Treat every surge as both a stress test and a learning opportunity, and invest in the tooling that makes the right response fast, simple, and safe.