Worth adding the operational half to this: a prompt is a production artifact, so it needs the same handling as code. Version it in the repo rather than editing it in a console, and keep a small fixed set of examples you rerun after every edit - otherwise the only way anyone discovers a prompt change was bad is a user complaint two weeks later. The smart-contract example makes another point implicitly. When the output feeds code rather than a person, the highest-leverage instruction is usually the output contract: give it a schema and validate against it, so a malformed response fails loudly at the boundary instead of flowing downstream as a plausible-looking value.
I should tighten my earlier wording: an all-green suite is useful, and the harness should still require green results for the behavior we already promise. The unresolved cases belong in a separate improvement set, with explicit expected behavior, rather than becoming failures everyone learns to ignore.
Then a prompt change can be checked against both questions: did it preserve the supported behavior, and did it resolve any known limitations? Once a limitation is fixed, promote that case into the regression suite. That keeps the signal without normalizing broken builds.
That’s a useful distinction. Separating known limitations from regression checks gives each set a clear purpose. Moving a case into the regression suite once it’s fixed also helps ensure the improvement is preserved through future prompt changes.
The promotion step is where an explicit expected result helps most. I would record the failing input, the behavior the fix is meant to change, and the prompt version that first passed it. Otherwise a later edit can preserve the example output while losing the property that motivated the case.
Keeping the limitation history beside the regression case also makes an intentional future behavior change distinguishable from accidentally weakening its assertion.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
Worth adding the operational half to this: a prompt is a production artifact, so it needs the same handling as code. Version it in the repo rather than editing it in a console, and keep a small fixed set of examples you rerun after every edit - otherwise the only way anyone discovers a prompt change was bad is a user complaint two weeks later. The smart-contract example makes another point implicitly. When the output feeds code rather than a person, the highest-leverage instruction is usually the output contract: give it a schema and validate against it, so a malformed response fails loudly at the boundary instead of flowing downstream as a plausible-looking value.