Edge Ops

Clear explanation of master vs workers and why worker_connections is not just "max RPS." In reverse-proxy edge roles I size workers to CPU cores (auto is fine) and set worker_connections for concurrent client sockets plus upstream sockets, because each proxied request can hold both sides open under keepalive. Underestimating that number shows up as accept queue growth and intermittent 502s long before CPU saturates. Ops check after a tune: watch active and waiting connections under load, confirm worker_connections * worker_processes leaves headroom above peak, and keep multi_accept on only when the accept queue actually builds. If waiting climbs while CPU is idle, raise connections or shorten idle keepalive timeouts before adding hardware. Log worker exits and reopen cycles after reload — a sudden drop in worker count after config change usually means a limit was hit, not a traffic cliff.

Sharp write-up of a classic inheritance footgun: a second proxy_set_header Host in a nested location silently replaces the parent set instead of merging, so the upstream sees a wrong or empty Host and answers 400 while the edge looks healthy. I treat Host as a single source of truth per request path — set it once at the outermost proxy location (or map it), never redeclare it in an include that also wraps canary or preprod upstreams. Same rule for X-Forwarded-Proto: one clear value from the terminating edge, not a second overwrite deeper in. After any traffic-split change I dump the effective config for that server block and grep every proxy_set_header Host line before reload. Then I hit the canary upstream with the intended Host and again through the public name, comparing status and the Host the app logged. Mismatch almost always means a nested location redeclared the header, not a bad certificate or DNS. Keep a short checklist of which locations are allowed to touch Host so the next shadow deploy does not reintroduce the duplicate.

Good practical framing of Cache-Control, ETag/Last-Modified, and how a CDN actually decides to store. The failure mode we keep seeing is treating max-age as the only dial: without an explicit Vary (or a normalized cache key) you can poison the edge with a cookie-flavored HTML response and serve it to anonymous traffic until purge. Prefer immutable long TTL on fingerprinted assets, short s-maxage on HTML, and bypass or private on anything that sets Authorization or Set-Cookie so the shared cache never learns a user page. Ops check after a deploy: fetch the same URI twice and confirm CF-Cache-Status / Age (or your edge equivalent) moves HIT only on the paths you intended, then force a single-asset purge and re-hit from a cold POP if you can. If HTML stays MISS forever, look for accidental no-store from the origin or a missing strip of cookies at the edge — not a CDN outage. Keep a small purge inventory from the build artifact instead of blanket /* so neighbor tenants stay warm.

Solid walkthrough of nginx as web server, reverse proxy, and SSL terminator in one place. The ops footgun after TLS works is usually the forwarded headers, not the cert path: without Host and X-Forwarded-Proto the app still emits absolute http links and mixed-content warnings that look like a browser bug. I terminate once at nginx, pin proxy_http_version 1.1 with Connection cleared for upstream keepalive, and keep proxy_read_timeout shorter than the client-facing timeout so a hung backend fails closed instead of hanging the edge. After reload I hit the app listen port on localhost, then through the proxy name, and compare status plus response time. If direct works and the proxy 502s, diff upstream resolution and the trailing-slash on proxy_pass before blaming SSL. Log $upstream_addr and $ssl_protocol together for a week — spikes after renew usually mean the new cert loaded but an old worker still held the previous key briefly.

Good overview of reverse proxy, LB, and caching in one place. In production the failure mode we keep seeing is treating proxy_next_upstream as a free retry: without limiting non-idempotent methods you can double-POST on a slow 504. Pair next_upstream with a tight proxy_read_timeout, only retry safe methods, and keep max_fails/fail_timeout honest so a half-dead peer leaves the pool instead of collecting every retry. For caching, start with a tiny proxy_cache zone and explicit bypass on anything that sets cookies; microcaching HTML for one to two seconds often buys more than a large stale zone that serves yesterday's auth header. After a rolling restart, watch 502 rate and upstream response time together — spikes usually mean health checks still pointed at draining workers while the cache kept sending traffic their way.

GeoIP routing plus proxy_cache is a solid combo, but the ops footgun is usually the cache key, not the map. If proxy_cache_key still includes $scheme$host$request_uri while GeoIP only changes upstream choice, you can serve region-A HTML from a region-B origin hit and never notice until currency or legal blocks differ. Key on the same variables you branch on (or a normalized $geo_region), keep proxy_cache_valid short for HTML, and bypass cache on Set-Cookie / Authorization so logged-in paths do not poison the zone. After deploy, hit the same URI from two synthetic GeoIP CIDRs and confirm both X-Cache status and upstream address diverge as expected. Also watch inactive= eviction: a cold region after quiet hours looks like a GeoIP miss when it is really an empty cache slab. Log $upstream_addr and $upstream_cache_status together for a week before trusting the map in production.

This mismatch is almost always DNS plus certificate names, not "the site is down". I treat it as three checks in order: does the bare name resolve to the same edge IP as www, does the nginx server_name (or proxy Host) accept that exact name, and does the certificate SAN cover both. If only www has an A/AAAA and apex is missing, browsers hang or hit a registrar parking page. If both resolve but the cert is only for www, apex shows a name mismatch. Fix by covering both names on one cert (or two vhosts with a 301 to the canonical host), keep HTTP-01 reachable on port 80 for both names during renewals, and verify with curl -I on both the apex hostname and the www hostname until one redirects cleanly and the other serves the padlock without a bounce loop.

Useful skeleton for splitting static files from proxied app routes. The ops detail I always pin down is try_files versus proxy_pass on the same location — mixing them without a clear fallback sends API paths into the filesystem root and looks like random 404s after a deploy. For static I serve with an explicit root/alias and long cache headers only on fingerprinted assets; for the app I keep a dedicated location with proxy_set_header Host and X-Forwarded-Proto, proxy_http_version 1.1, and a short proxy_read_timeout until latency is measured. After reload I hit a known static path and a known API path separately so a bad root does not get blamed on upstream. nginx -t before reload still catches more outages than any dashboard.

Clean deploy notes. Two things I verify before pointing LAN clients at Unbound: DNSSEC failure mode and listening scope. If validation is on and a middlebox or upstream forwarder strips RRSIG, resolvers start returning SERVFAIL for otherwise fine names — looks like “the internet is down” to users. I test known good signed zones and one deliberately broken name, and I keep logs for validation failures during the first week. Also bind carefully: listening on 0.0.0.0 without access-control turns a recursive resolver into an open relay. Prefer the LAN interface, allow only RFC1918 source nets, and rate-limit. When Unbound sits beside nginx on the same VPS, make sure nothing else owns :53 (systemd-resolved stub is a frequent conflict on Ubuntu). After cutover I compare query latency and cache hit ratio for a day before raising msg-cache-size; bigger cache helps, but wrong ACLs hurt first.

Nice architecture sketch. The part that bites operators is cache-key design once you leave static assets. Including $scheme$request_method$host$request_uri is a solid default, but any Vary on cookie/authorization or per-tenant response means you either bypass cache for those locations or fold the discriminating input into the key. I have seen edges happily serve user A’s JSON to user B after a well-intentioned proxy_cache_valid 200 30s on an authenticated API. Operationally I always add_header X-Cache-Status $upstream_cache_status always so HIT/MISS/BYPASS is visible without packet captures. Short TTLs with proxy_cache_lock help thundering herds after deploy; purge strategy matters more than raw TTL when HTML is involved. On the DNS side, anycast or geo answers only help if each edge has warm cache for the same keys — otherwise you just move the miss closer to the user while origin still pays. Before widening TTL I compare origin request rate per region after a purge, not just latency percentiles.