Discussion on "How I Investigate a Suspicious Login on a Linux Server"

Asep Sayyad · 2026-09-22T16:02:17.818Z

From auth.log and utmp to /proc memory forensics and backdoor persistence: a battle-tested triage playbook for production engineers. It is 2:14 AM. Your monitoring channel pings with a high-severity

This is one of the most thorough triage playbooks I've read here - the ctime-vs-mtime timestomping point and catching the pty-less sshd session via ss instead of w are both things most writeups skip entirely. One gap worth flagging for anyone running this against a cloud-hosted replica specifically: none of the persistence vectors here check the instance metadata service. If db-prod-01 is EC2, an attacker with even a few seconds of shell access can curl 169.254.169.254/latest/meta-data/iam/security-credentials/ and walk off with temporary IAM credentials that keep working long after you've frozen the box and rebuilt from known-good IaC - no local persistence needed at all. Worth checking VPC flow logs or CloudTrail for that request alongside the /proc and ld.so.preload checks (GuardDuty flags it directly as InstanceCredentialExfiltration), since the "rebuild the instance" advice at the end doesn't help if the stolen credentials are still valid off-box.

Great point. I focused mainly on host-level persistence in this playbook, but cloud credentials are an important part of the investigation too. Checking the instance metadata access and CloudTrail/GuardDuty activity should definitely be added for EC2 cases. Thanks for pointing this out!

Solid playbook, the ctime vs mtime timestomping point alone is worth bookmarking.

One thing I'd add right at the top, before any of the /proc work: the opening scenario mentions an SSH session "from an IP address block in a country where your engineering team has zero presence," but that judgment call is doing a lot of work and it's worth making it explicit rather than trusting the alert. A quick geolocation + ASN lookup on the source IP tells you fast whether you're looking at a residential ISP in an unexpected country (bad sign) or just a known corporate VPN exit / cloud NAT range that happens to geolocate oddly (false alarm). ipgeolocation.io's free tier is enough for this, one call gives you country, ASN and organization/network type, so you're not burning time on /proc forensics for what turns out to be a coworker on a company VPN.

Worth wiring straight into the alert itself so the country/ASN shows up before anyone even opens a terminal.

That's a good point. Country alone shouldn't be treated as proof of a suspicious login. Checking the IP's ASN and organization first can help separate a real anomaly from a VPN or cloud network. I'll definitely consider adding this to the initial triage steps. Thanks!

Thanks Asep. One more cheap triage step to put next to the ASN check: run the source IP against a threat feed before anyone opens a terminal. If it is already listed as an attacker or abuse source, you escalate faster. Disclosure, I work at WhoisFreaks, and our threat intelligence feed covers IP threats along with phishing and malware indicators, scored so you can load them into a SIEM or MISP. There are free samples to test with, and the full feed is a paid license: whoisfreaks.com/products/threat-intelligence-feed Any reputation list works for this step. Just treat a clean result as "no known history", not "safe".

Search Hashnode

How I Investigate a Suspicious Login on a Linux Server

Responses(5)