@itxcrusher

ArgoCD CVE-2022-24348: a Secret leak that hid in log volume

Jun 2 · 10 min read · The first thing we saw in Loki was a fanout service log line that contained the string 'a2V5Y2xvYWstY2xpZW50' repeated about 40 times in a single minute. Base64 decode: 'keycloak-client'. The fanout s

Why Grafana OnCall acknowledgments hang after a Helm upgrade migration

Jun 1 · 10 min read · The call did not come from our on-call rotation. It came from a customer who noticed two unrelated degradations on their side and asked why we had not paged. We had not paged because Grafana OnCall ha

Why a deleted backup Lambda kept billing 9,400 EBS snapshots

May 30 · 9 min read · The EBS Snapshot line on the monthly bill was $1,830. There was no active EBS snapshot policy on the account. The backup Lambda that had produced these snapshots had been deleted thirteen months earli

Why one shared Terraform module made every PR a 14-service change

May 26 · 10 min read · The PR that shipped the bug had three approvals and a comment that read "LGTM, plans look normal." The plans were not normal. They were 14 separate terraform plan outputs stacked in the CI log, each t

Why a forgotten RDS replica added $8,600 to one AWS bill

May 25 · 8 min read · The finance lead forwarded the AWS bill on a Monday morning with three question marks in the subject line. The number had gone from a steady $3,200/month to $11,800 in six days. The on-call engineer