How a do-not-disrupt annotation broke Karpenter consolidationOur Karpenter cluster went from 108 nodes to 198 over one weekend while pod count barely moved, consolidation fell to about an eighth of its normal rate, and our EC2 on-demand bill jumped 2.2x. The ca16 min read·blog.infraforge.agency1Aug 18
Worker drops jobs intermittently: startup race, env drift, schema skewIf your worker is dropping jobs intermittently, sometimes crashing at boot with 'Error 111 connecting to redis:6379. Connection refused' and sometimes producing an empty result.json with no error at a18 min read·blog.infraforge.agencyAug 10
How a missing S3 gateway endpoint route quintupled our NAT billThe finance lead asked why AWS charged us $2,100 for NAT gateway data processing last month. Our normal was around $400. Nothing in the release calendar explained it: no new services, no traffic bump11 min read·blog.infraforge.agency1Jul 16
How one stuck PDB doubled our EKS autoscaler bill in six daysThe EC2 bill for our EKS cluster went from $4,200 to about $8,600 over six days, against the same six days the month before. Same cluster, no launch, no traffic event, no team asking for headroom. kub15 min read·blog.infraforge.agencyJul 16
Recovering a status page from a half-finished schema migrationThe log line was 'database schema version 23 is dirty, refusing to start' and the pod exited immediately after printing it. The team had already tried a Helm rollback to the previous chart version. Th15 min read·blog.infraforge.agencyJul 13