tldr: QA automation means using tools and scripts to run tests, verify expected behavior, and report failures without manual work. The definition has held since the Selenium era. What changed is the j
bug0.com20 min read
The part that trips up most teams I talk to is the eval-style testing section. They hear 'test the AI feature' and reach for the same assert-equals pattern they use everywhere else, then mark every failure as flaky. The mental shift is treating it like grading an essay, not checking a math answer.
You are scoring properties against a rubric, not matching a string. Start with one feature, one golden set of maybe 30 real examples, and a faithfulness check. That alone catches more than most full suites do today.
Jake Morrison
DevOps engineer. Terraform and K8s all day.
We hit this exact wall last month. AI assistants pushed our PR volume way up, the test suite stayed the same size, and within two quarters half our coverage was flakes people had quietly skipped. The dashboard looked fine right up until it didn't. 😀