"Saturation is not an error" should be printed on dashboards. Error-rate alerts are blind to a pool quietly running out of workers because nothing ever fails. After this one, what did you alert on - busy workers as a share of pm.max_children, or the listen queue?
Thanks Abhinav. Listen queue first: any value above 0 for more than a few seconds means requests are already waiting. Busy workers above 70 percent of pm.max_children is the early warning. Both come from the PHP-FPM status page. Error-rate alerts never see this, which is why the team spent two days looking for an attacker.
That 70 percent early-warning line is the part most setups skip. By the time the queue shows anything above zero, users already feel it. Thanks for the numbers.
Abhinav Sharma
Founder of Impact Gate. See who breaks before you change an API.
"Saturation is not an error" should be printed on dashboards. Error-rate alerts are blind to a pool quietly running out of workers because nothing ever fails. After this one, what did you alert on - busy workers as a share of pm.max_children, or the listen queue?