---
title: "Discussion on \"The Ghost in the Apache: Debugging a Mystery Cron Job Across 250+ Legacy EC2 Instances\" | Hashnode"
description: "Discussion on \"The Ghost in the Apache: Debugging a Mystery Cron Job Across 250+ Legacy EC2 Instances\". Hidden hourly script was undoing our fixes, and it had been waiting in a Golden AMI for years."
url: "https://hashnode.com/posts/the-ghost-in-the-apache-debugging-a-mystery-cron-job-across-250-legacy-ec2-instances/6ae725bc713143a41082216c"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [The Ghost in the Apache: Debugging a Mystery Cron Job Across 250+ Legacy EC2 Instances](https://smahima.hashnode.dev/the-ghost-in-the-apache-debugging-a-mystery-cron-job-across-250-legacy-ec2-instances?utm_source=hashnode&utm_medium=feed)

Hidden hourly script was undoing our fixes, and it had been waiting in a Golden AMI for years.

Nov 1 · 8 min read · smahima.hashnode.dev  
Read the article

-   [#devops](https://hashnode.com/tag/devops)
-   [#sre](https://hashnode.com/tag/sre)
-   [#aws](https://hashnode.com/tag/aws)
-   [#security](https://hashnode.com/tag/security)
-   [#incident-response](https://hashnode.com/tag/incident-response)
-   [#linux](https://hashnode.com/tag/linux)
-   [#cronjob](https://hashnode.com/tag/cronjob)

## Written by

[Mahima Singh](https://hashnode.com/@smahima)

@smahima · Hyderabad, India

  
[View profile](https://hashnode.com/@smahima)

## Responses 1

Markdown supported · ⌘↵ to post

[Ascendra Ventures](https://hashnode.com/@ascendra-ventures)

[Nov 2](https://hashnode.com/posts/the-ghost-in-the-apache-debugging-a-mystery-cron-job-across-250-legacy-ec2-instances/6ae725bc713143a41082216c/comment/6ae7f7605616d8f395065c3f)

The fix that comes back twenty minutes later is the worst kind of incident, because it makes everyone doubt the diagnosis that was actually right. Finding that a script from the golden AMI was restoring the old Apache config explains why every instance behaved the same way.

Your note that crontab -l only shows one user's crontab is worth repeating to every on-call engineer. Checking /etc/cron.d, /var/spool/cron and systemd timers takes a minute, and it would have saved most of the 35 minutes of tracing.

The broader lesson for me is the inventory of what gets baked into base images. Legacy fleets collect automation nobody remembers writing, and a CSP change from a new security agent is exactly the kind of thing that wakes it up.

When Ascendra Ventures takes on DevOps work on older estates, an audit of scheduled jobs is one of the first deliverables. Did you remove the script from the AMI or replace it with managed config?