---
title: "Discussion on \"Run forty experiments against one eval set and you will find an improvement that is not there\" | Hashnode"
description: "Discussion on \"Run forty experiments against one eval set and you will find an improvement that is not there\". You iterate on a prompt, run the eval set, keep the best variant. Repeat for a quarter. The number that survives is the maximum of forty noisy measurements, and the maximum of noisy measurements is an"
url: "https://hashnode.com/posts/run-forty-experiments-against-one-eval-set-and-you-will-find-an-improvement-that-is-not-there/6a7c7e4cae9b7572d6527799"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Run forty experiments against one eval set and you will find an improvement that is not there](https://llmasajudge.hashnode.dev/run-forty-experiments-against-one-eval-set-and-you-will-find-an-improvement-that-is-not-there?utm_source=hashnode&utm_medium=feed)

You iterate on a prompt, run the eval set, keep the best variant. Repeat for a quarter. The number that survives is the maximum of forty noisy measurements, and the maximum of noisy measurements is an

Aug 12 · 5 min read · llmasajudge.hashnode.dev  
Read the article

-   [#llm-evaluation](https://hashnode.com/tag/llm-evaluation)
-   [#statistics](https://hashnode.com/tag/statistics)
-   [#data-science](https://hashnode.com/tag/data-science)
-   [#experimentation](https://hashnode.com/tag/experimentation)

## Written by

[mayaandersson](https://hashnode.com/@mayaanderssondev)

@mayaanderssondev · Palo Alto CA

  
[View profile](https://hashnode.com/@mayaanderssondev)

Just a bored curious dev

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
