A $3,900 overnight bill from our LLM eval suite: the incident, and the spend guard I shipped after

TL;DR. Our LLM-judge eval suite had no cost ceiling. It ran the full judge over 1,200 cases on every CI trigger. On 08/07 a dependency bot opened 41 pull requests between 01:00 and 04:00, and our merg