The API Wasn't Slow: Finding a 2-Minute Delay in the Delivery Path

The Nginx timing split is the part I would steal for every team: upstream response time under 300 ms against a total request time of almost 80 seconds makes it obvious the API never had the problem, and once both numbers are logged side by side, every future "slow API" report becomes a one-line check. One suspect I would add weight to is the load generator itself — when the generator runs short on CPU or bandwidth, the failure pattern looks exactly like a flaky network, and one in three requests failing rather than all of them slowing down fits that shape. Did you get a chance to check the generator's own resource usage while the test was running?

The outside-vs-server replay is the cleanest part of this diagnosis. If the VPN is the suspect, I would run iperf3 across the tunnel during a load test and compare response size times requests per second with what the tunnel can actually carry. That usually separates a bandwidth limit from an MTU problem quickly.

The outside-vs-server replay is a clean way to narrow the fault domain, and comparing Nginx upstream time with total request time makes the mismatch concrete. One nuance: request_time minus upstream_response_time is not necessarily just response delivery; it can also include receiving the request body and proxy queueing. Logging request_length and body_bytes_sent alongside the timings could help distinguish a slow upload from a slow response path. Did response size vary across the failing runs?