The file-nr part is the one I would spell out in a runbook. On 2.6 and later, /proc/sys/fs/file-nr is three columns: allocated handles, a free-handles field that is hardwired to 0 (allocated already equals in use, so the middle number is not spare capacity), and the max. Reading column 2 as free is how people either bump file-max for nothing or miss a process that is only hitting its own RLIMIT_NOFILE.
Also, lsof -p PID and ls /proc/PID/fd | wc -l will not match even when nothing is leaking. lsof includes cwd, root, and mapped files that are not fds. If the count is climbing, group the symlinks under /proc/PID/fd first (socket vs pipe vs a path ending in (deleted)) before touching LimitNOFILE. A log the process still holds after rotation is a different fix from a socket that never closes.
c-wind
Building developer tools around SSH and terminals. Rust, Linux, ops.
The file-nr part is the one I would spell out in a runbook. On 2.6 and later, /proc/sys/fs/file-nr is three columns: allocated handles, a free-handles field that is hardwired to 0 (allocated already equals in use, so the middle number is not spare capacity), and the max. Reading column 2 as free is how people either bump file-max for nothing or miss a process that is only hitting its own RLIMIT_NOFILE.
Also,
lsof -p PIDandls /proc/PID/fd | wc -lwill not match even when nothing is leaking. lsof includes cwd, root, and mapped files that are not fds. If the count is climbing, group the symlinks under /proc/PID/fd first (socket vs pipe vs a path ending in (deleted)) before touching LimitNOFILE. A log the process still holds after rotation is a different fix from a socket that never closes.