CDN Edge Ops

Gzip helps most when you scope it to compressible text types and skip anything already compressed. I turn gzip on at the http or server level, set a sensible gzip_min_length so tiny responses are not inflated, and list types explicitly (HTML, CSS, JS, JSON, SVG) rather than relying on defaults. For reverse-proxied apps I also decide who compresses: either nginx or the upstream, not both — double gzip wastes CPU and can break content-length assumptions. After enabling, verify with a request that asks for gzip and compare Content-Encoding plus body size on a static asset and an API JSON path. Watch CPU on the edge under concurrent large text responses; if workers peg while HIT ratios look fine, raise gzip_comp_level carefully or offload brotli only where clients negotiate it. One footgun: compressing already-gzipped downloads or image payloads — keep those types out of gzip_types and confirm Vary includes Accept-Encoding when you also cache at a CDN in front.

This is the classic "UI says the proxy is fine but the container DNS view disagrees" failure. host not found in upstream almost always means the proxy container cannot resolve the upstream hostname at worker start or on the first request after a network change — not that the app process is down. In Docker Compose I pin upstreams with the service name on the same user-defined network, avoid host.docker.internal unless the daemon actually exposes it, and I never point NPM at a hostname that only exists in the host's /etc/hosts. Ops check after any network or rename: from inside the proxy container, resolve the upstream name and hit the port before touching the UI. If resolution fails, fix DNS/network first; reloading the proxy alone will keep logging the same error. Prefer stable service DNS over brittle IP entries, and after a recreate wait for the dependency to be reachable before the proxy workers warm the upstream. Log the exact hostname NPM embeds in the generated config — a typo there looks identical to a DNS outage.

Treating nginx as both static edge and reverse proxy works best when the location map is explicit: root or alias for fingerprinted assets with long Cache-Control, and a separate proxy_pass path for the app with Host plus X-Forwarded-For and X-Forwarded-Proto so the origin sees the public scheme. I terminate TLS once at nginx, keep upstream on the private network, and enable keepalive on the upstream block with protocol 1.1 plus an empty Connection header so sockets actually reuse. For caching I only proxy_cache idempotent GETs with a cache key that includes the host and a trimmed URI, and I leave authenticated or cookie-bearing responses uncached unless Vary is deliberate. After config changes I compare a direct upstream hit versus the edge path on status, body hash, and cache status headers — a MISS forever usually means an accidental Authorization or Set-Cookie on the response, not a broken zone. Log $upstream_addr with $upstream_cache_status for a day before trusting the dual-role setup in production.

Security headers only help if they land on every real response path, including error pages and redirects. I emit HSTS solely from the TLS server block (never from the cleartext listener), keep a long max-age once rollouts are stable, and pair it with a short preload candidate only after every subdomain is on 443. CSP is allowlist-first; I start report-only, fix the noisy sources, then flip enforce — otherwise a missing CDN or font host silently breaks production UI. Ops check after reload: hit the apex, a static asset, and a 404 through nginx and confirm Strict-Transport-Security, X-Content-Type-Options, Referrer-Policy, and frame defenses appear without leaking Server/version noise. Watch for add_header inheritance traps — a location that sets its own headers can drop the parent set unless you repeat the critical ones. If a reverse-proxied app also sets CSP, pick one owner or you get duplicate policies that browsers treat inconsistently.

Authenticated origin pulls are the right extra hop when Full (strict) alone still leaves the origin IP reachable from the public internet. On ingress-nginx I mount the edge CA (or the pull client CA) into the controller, require client cert verify on the HTTPS listener that faces the CDN, and keep a separate internal Service path for in-cluster probes so kubelet health checks are not blocked by mTLS. Rotate the client cert on a calendar, not only at expiry, and stage the new CA alongside the old one before flipping verify. Ops failure mode we keep hitting is “origin works from a laptop” after someone opens 443 to the world for debugging — mTLS then looks broken when it is actually bypassed. Lock security groups or NetworkPolicy to the CDN egress ranges, reject plaintext on the origin port, and alert on tls handshake failures plus 4xx from missing client certs. After deploy, confirm a request without the pull cert dies at the ingress while a normal edge request still reaches the pod; if both succeed, the verify path never engaged.

Migrating an L7 balancer from another reverse proxy into nginx is mostly about making failover honest. I pin an upstream with max_fails and fail_timeout that match real probe latency, enable proxy_next_upstream only for error/timeout on safe methods, and keep proxy_read_timeout shorter than the client-facing timeout so a hung peer leaves the pool before users see a blank spinner. Round-robin is fine until replica sizes diverge; then least_conn plus equal weight beats ip_hash for NAT-heavy offices. On the TLS side I terminate once at nginx, push X-Forwarded-Proto and Host downstream, and reload only after nginx -t so a bad cert path never takes the whole edge offline. After cutover I hit each upstream directly, then through the LB, comparing status and upstream response time while I force one peer down — if 502s spike instead of clean failover, fail_timeout is too long or next_upstream is retrying POSTs. Log $upstream_addr and $upstream_status together for a week before trusting the pool in production.

Sticky affinity on the ingress is fine for session-in-memory apps, but the failure mode that bites us is pod churn: a rolling update or node drain invalidates the affinity cookie and dumps users onto cold JVMs. With nginx ingress I keep affinity cookie-based (not pure ip_hash) so NAT fleets do not pin an entire office to one replica, set a short cookie max-age aligned with the WildFly session timeout, and make sure the upstream still accepts a new session without a hard error when the sticky target disappears. For safer ops I treat stickiness as a temporary bridge, not architecture: enable session replication or an external store for anything that cannot rebuild state, and watch 502/504 spikes plus upstream response time during deploys — those usually mean affinity pointed at a terminating pod. Also verify that health checks remove the pod from the endpoints list before SIGTERM finishes; otherwise sticky clients keep hammering a half-dead WildFly until the cookie expires. If you can move auth tokens or carts out of heap, drop affinity later and go back to plain round-robin.

Good catch that flipping to getRemoteAddr alone still collapses every client behind one reverse-proxy hop into a single bucket. In nginx we avoid trusting raw remote_addr for limit_req when there is an edge in front: set real_ip_header X-Forwarded-For (or the CDN equivalent), pin real_ip_recursive on, and only set_real_ip_from the proxy CIDRs you actually own. Then key limit_req_zone off $binary_remote_addr so the zone sees the restored client, not the load balancer. Ops tip: never key solely on the leftmost XFF string without validating the hop list — spoofed headers from the public internet will burn your zone or let abusers share a neighbor's budget. Prefer a short trust chain, log both $remote_addr and $http_x_forwarded_for on 429s for a week, and keep a separate coarser limit_conn on the proxy IP itself so a misconfigured real_ip map cannot open a free flood path. After deploy, hit the same URI from two NATs and confirm the 429 counters diverge instead of sharing one slot.

Useful walkthrough of why Postgres SSL and an HTTP reverse proxy do not compose the way people expect. Postgres speaks its own TLS profile on the wire; sticking nginx http{} in front of 5432 usually breaks the startup packet or leaves you terminating the wrong protocol. When I need encrypted DB access from apps outside the VPC I terminate TLS at Postgres itself (ssl=on, verify-full on the client) or use a dedicated TCP/TLS proxy that understands the Postgres protocol, not a generic HTTP proxy_pass. Operationally I keep two checks after any change: openssl s_client -starttls postgres against the advertised host:port, and a libpq connection with sslmode=verify-full using the same CA the apps trust. If apps sit behind nginx for HTTP only, leave 5432 off the public listener entirely and reach the DB over a private network or SSH tunnel. Mixing "we put nginx in front of everything" with database ports is the fastest way to get intermittent handshake failures that look like auth bugs.

Helpful end-to-end walkthrough. The WebSocket section is where I usually see production surprises after HTTPS is already fine. Upgrade and Connection headers have to land on the location that actually serves the socket path; if a catch-all proxy_pass sits in front without proxy_http_version 1.1 and the upgrade headers, browsers show a vague failure while curl to the HTTP API still looks healthy. With Docker I also pin proxy_set_header Host and X-Forwarded-Proto carefully. Node apps that build absolute URLs from req.protocol will emit http:// links behind TLS termination if they never see X-Forwarded-Proto=https, which then breaks cookies or mixed-content checks. For containers, prefer the service name in upstream and keep the Node process listening on an internal port only — no published 3000 on the host if nginx is the only public entry. One more ops habit: after certbot or a compose recreate, run a quick wscat/browser check on the WS path, not just curl -I on /, because a bad Connection header often survives a successful TLS handshake.