---
title: "Discussion on \"Treating LLM prompts like code: a regression catalog for AI failures\" | Hashnode"
description: "Discussion on \"Treating LLM prompts like code: a regression catalog for AI failures\". A field note on turning prompt-engineering folklore into structured, regression-tested artifacts."
url: "https://hashnode.com/posts/treating-llm-prompts-like-code-a-regression-catalog-for-ai-failures/6a0d16f88837277411c2e65e"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Treating LLM prompts like code: a regression catalog for AI failures](https://reconpe.hashnode.dev/treating-llm-prompts-like-code-a-regression-catalog-for-ai-failures?utm_source=hashnode&utm_medium=feed)

A field note on turning prompt-engineering folklore into structured, regression-tested artifacts.

May 20 · 10 min read · reconpe.hashnode.dev  
Read the article

-   [#llm](https://hashnode.com/tag/llm)
-   [#promptengineering](https://hashnode.com/tag/promptengineering)
-   [#software-architecture](https://hashnode.com/tag/software-architecture)
-   [#artificial-intelligence](https://hashnode.com/tag/artificial-intelligence)

## Written by

[Amit Mishra](https://hashnode.com/@amitreconpe)

@amitreconpe · India

  
[View profile](https://hashnode.com/@amitreconpe)

Fintech and AI

A curious engineer simplifying engineering.

## Responses 2

Markdown supported · ⌘↵ to post

[ethanwalker](https://hashnode.com/@ethanwritesai)

[May 21](https://hashnode.com/posts/treating-llm-prompts-like-code-a-regression-catalog-for-ai-failures/6a0d16f88837277411c2e65e/comment/6a0eae358b03460221015d5d)

Treating prompts like code is the right framing. We added a CI hook with Promptfoo that blocks any merge where regression-test scores drop more than 5%. The hardest part wasn't writing the eval set, it was getting the team to maintain it as prompts evolved. Curious if your catalog covers the silent-degradation case where prompts pass eval but drift in real-world distribution.

[Amit Mishra](https://hashnode.com/@amitreconpe)

[May 21](https://hashnode.com/posts/treating-llm-prompts-like-code-a-regression-catalog-for-ai-failures/6a0d16f88837277411c2e65e/comment/6a0eae358b03460221015d5d#reply-6a0eba668b034602210b0a5d)

The catalog is a registry of failure modes we've already seen, each one pinned by an anti-pattern rule and a JUnit lock test. That's a regression net, it catches a known failure recurring, not a new one. Tests run fixed inputs, so drift slips past them. This is a must in our workflow as well if our developers are using claude, codex or anything its a signal to the agentic coding platform as well.

We catch drift production-side instead. Every mapping-suggestion call writes an adherence score, and we log the gap between what the AI proposed and what the user actually saved, by domain. When users keep overriding the AI in one domain, that's drift showing up in the data.

The honest gap is the alert (we are making this work in right way in our upcoming sprint). Right now a human has to read the telemetry dashboard to catch it, a threshold that fires on an adherence drop is what's still missing.
