---
title: "ethanwalker (@ethanwritesai) | Hashnode"
description: "ethanwalker (@ethanwritesai) writes on Hashnode. Read their latest articles and follow their work."
url: "https://hashnode.com/@ethanwritesai"
---

@ethanwritesai

# ethanwalker

@ethanwritesai · San Francisco, USA · Joined May 2026

### [A 91% eval pass rate shipped our worst regression. We gate on the delta now.](https://ethanwalkerwrites.hashnode.dev/a-91-eval-pass-rate-shipped-our-worst-regression-we-gate-on-the-delta-now?utm_source=hashnode&utm_medium=feed)

Our CI eval gate failed anything under 90%. A change landed at 91%, passed, and shipped a regression that cost us two days. The absolute threshold was the problem. it cannot tell a stable 91% from a 9

3 min read · [ethanwalkerwrites.hashnode.dev](https://ethanwalkerwrites.hashnode.dev/a-91-eval-pass-rate-shipped-our-worst-regression-we-gate-on-the-delta-now?utm_source=hashnode&utm_medium=feed)

[Aug 17](https://hashnode.com/posts/a-91-eval-pass-rate-shipped-our-worst-regression-we-gate-on-the-delta-now/6a834030b5087402bc9ddf30)

### [356 of the 528 eval cases we could judge have never failed. Our pass rate could only move 21 points.](https://ethanwalkerwrites.hashnode.dev/356-of-the-528-eval-cases-we-could-judge-have-never-failed-our-pass-rate-could-only-move-21-points?utm_source=hashnode&utm_medium=feed)

TL;DR: I broke out per-case results for the incident-harvested part of our eval suite, 611 of our 1,400 cases, and asked which cases have ever discriminated between two shipped versions. Of the 528 wi

8 min read · [ethanwalkerwrites.hashnode.dev](https://ethanwalkerwrites.hashnode.dev/356-of-the-528-eval-cases-we-could-judge-have-never-failed-our-pass-rate-could-only-move-21-points?utm_source=hashnode&utm_medium=feed)

[Aug 13](https://hashnode.com/posts/356-of-the-528-eval-cases-we-could-judge-have-never-failed-our-pass-rate-could-only-move-21-points/6a7e1b263c3b97238af3dfed)

### [We back-tested our eval gate against 41 known regressions. It caught 23](https://ethanwalkerwrites.hashnode.dev/we-back-tested-our-eval-gate-against-41-known-regressions-it-caught-23?utm_source=hashnode&utm_medium=feed)

The gate had been green for eleven weeks. In the same eleven weeks we rolled back two prompt changes, hotfixed a retrieval config on a Saturday, and shipped a summarizer that started dropping the seco

10 min read · [ethanwalkerwrites.hashnode.dev](https://ethanwalkerwrites.hashnode.dev/we-back-tested-our-eval-gate-against-41-known-regressions-it-caught-23?utm_source=hashnode&utm_medium=feed)

[Aug 11](https://hashnode.com/posts/we-back-tested-our-eval-gate-against-41-known-regressions-it-caught-23/6a7b52887f5926cb10211e7f)

### [Our eval gate runs 22 minutes. The queue behind it hit three hours.](https://ethanwalkerwrites.hashnode.dev/our-eval-gate-runs-22-minutes-the-queue-behind-it-hit-three-hours?utm_source=hashnode&utm_medium=feed)

Nobody complained about the gate. They complained about Tuesday. Our merge queue runs the full eval suite before anything lands: 1,400 cases, about 240 of them scored by an LLM judge, the rest determi

6 min read · [ethanwalkerwrites.hashnode.dev](https://ethanwalkerwrites.hashnode.dev/our-eval-gate-runs-22-minutes-the-queue-behind-it-hit-three-hours?utm_source=hashnode&utm_medium=feed)

[Aug 6](https://hashnode.com/posts/our-eval-gate-runs-22-minutes-the-queue-behind-it-hit-three-hours/6a74be835737f92c017ca8f2)

### [Half my eval regressions never touched the prompt file](https://ethanwalkerwrites.hashnode.dev/half-my-eval-regressions-never-touched-the-prompt-file?utm_source=hashnode&utm_medium=feed)

I keep a tally. Over the last two quarters, a little more than half of the output regressions I traced in one RAG service came from a change that never went near the prompt directory. The eval gate wa

8 min read · [ethanwalkerwrites.hashnode.dev](https://ethanwalkerwrites.hashnode.dev/half-my-eval-regressions-never-touched-the-prompt-file?utm_source=hashnode&utm_medium=feed)

[Aug 4](https://hashnode.com/posts/half-my-eval-regressions-never-touched-the-prompt-file/6a723038a3641fd5631f5b08)
