Jake Morrison

The false green section hits. We had a retry-based setup mask a broken coupon flow for 11 days because the test kept passing on the second attempt. Nobody looks at a green suite. Support tickets found the bug before CI did. Since then I trust a loud failure over a quiet pass every time.

Most QA pitches answer none of these. They lead with coverage percentages and test counts, which are engineering metrics, and test count in particular is a metric that rewards waste. The pitch that works prices the status quo first. Everything below uses one worked example, a 12-engineer team at a fully loaded $75 an hour, so you can swap in your own numbers line by line. Couldn't agree more!

Good writeup but I think you're underselling how annoying the CLI bookkeeping gets at scale. Session names and disk cleanup add up fast when you're running 30 flows in CI. The Passmark caching angle is interesting though, hadn't thought about the maintenance cost being the real bill until now.

The "run the same set through all of them" advice is spot on, wish I'd done that before we signed a seat-based contract that got expensive fast. Curious where you'd put the line between misc steel a tool should handle vs what's still on the estimator?

We hit this exact wall last month. AI assistants pushed our PR volume way up, the test suite stayed the same size, and within two quarters half our coverage was flakes people had quietly skipped. The dashboard looked fine right up until it didn't. 😀

Playwright based automation is everywhere in the AI era. Love how the domain is shaping up. Looks like Playwright is the optimal framework to chose when investing in-house or managed.

Nice write up, Sushrut.

Serious apps built using Lovable will care about QA a lot more than side projects.

Good one, Fazle!

the local chromium ceiling section is exactly what we hit last year. started with 3-4 parallel workers on a self-hosted runner, everything fine. scaled to 15 and tests started timing out randomly. spent two weeks debugging "flaky selectors" before realizing the runner was just OOMing silently. the one-line swap from launch to connectOverCDP is the part i wish someone had told me earlier. in my head "moving to cloud browsers" sounded like a weekend migration project. ended up being a 30 minute change. one question though - for long-running agent loops, how are folks handling session cleanup when the agent crashes mid-run? curious if providers auto-terminate or if you eat the idle time until timeout.