---
title: "Discussion on \"Why typed decisions?\" | Hashnode"
description: "Discussion on \"Why typed decisions?\". Getting an LLM to return a value your code can branch on with a confidence you can act on"
url: "https://hashnode.com/posts/why-typed-decisions/6ad2ac5aa305ffe6691817a4"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Why typed decisions?](https://liminma.hashnode.dev/why-typed-decisions?utm_source=hashnode&utm_medium=feed)

Getting an LLM to return a value your code can branch on with a confidence you can act on

Oct 16 · 14 min read · liminma.hashnode.dev  
Updated Oct 25  
Read the article

-   [#llm](https://hashnode.com/tag/llm)
-   [#python](https://hashnode.com/tag/python)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#ai](https://hashnode.com/tag/ai)

## Written by

[Limin Ma](https://hashnode.com/@liminma)

@liminma · Toronto, Canada

  
[View profile](https://hashnode.com/@liminma)

## Responses 4

Markdown supported · ⌘↵ to post

[Rose Alvina](https://hashnode.com/@rosealvinanex)

[Oct 17](https://hashnode.com/posts/why-typed-decisions/6ad2ac5aa305ffe6691817a4/comment/6ad2ff3c3a18711331186aa7)

Nice framing of act / clarify / hand off as the real output. A practical note for anyone applying this to a support bot: pick the two thresholds from costs rather than accuracy. Estimate what a wrong automatic action costs versus a clarifying question versus a human handoff, then choose the act and hand-off cutoffs on a held-out set that minimise total expected cost. That usually lands you on a high "act" threshold and a surprisingly wide "clarify" band. Also watch calibration drift: track the share of traffic in each band per week. If the "act" share drops suddenly, new intents or wording have appeared, and that is the moment to label a fresh batch and refit, which as you note takes seconds. Finally, when an intent's top two labels are close in probability, a clarifying question that names both options tends to resolve it in one turn.

[Limin Ma](https://hashnode.com/@liminma)

[1d ago](https://hashnode.com/posts/why-typed-decisions/6ad2ac5aa305ffe6691817a4/comment/6ad2ff3c3a18711331186aa7#reply-6af631fe0c601f44a67f392c)

Thanks for the practical notes. [Post 8](https://liminma.hashnode.dev/from-confidence-to-action) starts from the same three costs but turns them into two targets instead: actions should be right at least 98% of the time, and questions should offer the right intent at least 90% of the time. The result is what you predicted, with a high act threshold and 39.9% of test messages in the ask band. Minimizing expected cost directly is cleaner when a team can say roughly how the outcomes compare.

Tracking weekly band shares is a nice addition. The post only suggests checking how often actions are right, which needs labels. Band shares don't, so they can flag drift sooner.

On your last point: your tip on close pairs fills a gap in the post. When two intents are close, the confidence is at most about 0.5 and often lower. Many of these answers fall below the ask threshold of 0.493 and go to a human agent. Your rule would offer them the two options instead. I haven't tested it, so I can't say how many answers it would catch.

[StackShortlist](https://hashnode.com/@veristria)

[Oct 17](https://hashnode.com/posts/why-typed-decisions/6ad2ac5aa305ffe6691817a4/comment/6ad2ee7d297ccb0383ba4979)

Being upfront about the trade is what makes this series credible: 86.9% on the BANKING77 test set, compared against 83 to 85% for classifiers trained on 10 messages per intent and about 93% with full training data. Framing the goal as knowing which answers to act on, which to clarify and which to hand off is a practical way to think about it. Refitting the calibration in seconds when labels change, instead of retraining, also keeps the setup cheap to maintain.

[Limin Ma](https://hashnode.com/@liminma)

[1d ago](https://hashnode.com/posts/why-typed-decisions/6ad2ac5aa305ffe6691817a4/comment/6ad2ee7d297ccb0383ba4979#reply-6af6365ac2b68ec6d70cdb84)

Thanks for reading. [Post 8](https://liminma.hashnode.dev/from-confidence-to-action) is now up. It shows how the thresholds are chosen and how much the 98.7% could vary on other messages. One detail this post leaves out: the model needs a few minutes to answer the new calibration messages before the refit after a label change.