---
title: "Discussion on \"An ML search loop that can't overfit its own eval — the gate is in code, not a prompt\" | Hashnode"
description: "Discussion on \"An ML search loop that can't overfit its own eval — the gate is in code, not a prompt\". I kept seeing agent and eval demos where the honesty — held-out discipline, \"no metric gaming\" — lives in a prompt, or in a paper's methodology section. So I tried to build the opposite: a search loop"
url: "https://hashnode.com/posts/an-ml-search-loop-that-can-t-overfit-its-own-eval-the-gate-is-in-code-not-a-prompt/6a64a424431a612521b53833"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [An ML search loop that can't overfit its own eval — the gate is in code, not a prompt](https://jd-dev.hashnode.dev/an-ml-search-loop-that-can-t-overfit-its-own-eval-the-gate-is-in-code-not-a-prompt?utm_source=hashnode&utm_medium=feed)

I kept seeing agent and eval demos where the honesty — held-out discipline, "no metric gaming" — lives in a prompt, or in a paper's methodology section. So I tried to build the opposite: a search loop

Jul 25 · 3 min read · jd-dev.hashnode.dev  
Read the article

-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#ai](https://hashnode.com/tag/ai)
-   [#opensource-inactive](https://hashnode.com/tag/opensource-inactive)
-   [#AI](https://hashnode.com/tag/AI)
-   [#machi](https://hashnode.com/tag/machi)

## Written by

[Jeet Dhandha](https://hashnode.com/@jeetdhandha)

@jeetdhandha

  
[View profile](https://hashnode.com/@jeetdhandha)

## Responses 1

Markdown supported · ⌘↵ to post

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Jul 25](https://hashnode.com/posts/an-ml-search-loop-that-can-t-overfit-its-own-eval-the-gate-is-in-code-not-a-prompt/6a64a424431a612521b53833/comment/6a6539c5e4ccbd81f7fe7bef)

Putting the gate in code is much stronger than putting honesty in a prompt, but repeated held-out feedback is still adaptive access. Even if generated code cannot read the grader, the search policy observes scores and pass or kill outcomes across nodes, which can eventually tune itself to that slice. I would use dev for search, a query-budgeted validation set for ratcheting, and one locked final test used only once, with the ledger recording every validation access.
