The outside-vs-server replay is the cleanest part of this diagnosis. If the VPN is the suspect, I would run iperf3 across the tunnel during a load test and compare response size times requests per second with what the tunnel can actually carry. That usually separates a bandwidth limit from an MTU problem quickly.
Rishabh Mishra