Discussion on "Your MCP tools never told the agent what they can do"

JAY TANK · 2026-10-06T03:22:28.975Z

There is a moment, early in wiring up an MCP server, where you stop thinking about the tools and start thinking about the feature. You had a clean plan - this server exposes an invoice API to an agent

Skipping the vector database for the first chatbot is the right call. The failure I keep seeing is retrieval without a refusal: the model answers from the top hit even when the hit is a weak match. What score do you require before you answer, and what do you say when nothing clears it? iin1006h19

Exactly - retrieval without a refusal is the quiet failure mode: the model treats the top hit as an answer no matter how weak the match is. My rule of thumb is two gates, not one: a minimum similarity score below which nothing is eligible, and a margin check (the top hit must beat the runner-up by a set gap, so "lots of mediocre matches" doesn't sneak through). A concrete starting point for cosine on normalized embeddings: require ~0.75+ to answer, treat 0.6–0.75 as "retrieve but hedge," and refuse below that - then tune on your own eval set, because the right threshold is corpus-specific.

When nothing clears the bar, the honest move is a refusal with a next step, not a guess: "I don't have a confident source for that," plus what I do have (closest topics) or a handoff. A visible "no answer" is a feature —-it's the line between a bot that's confidently wrong and one people actually trust.

The annotation is the permission model, not a docstring. If the tool description is missing, the agent will guess the argument shape and the server will accept a wider call than the human would. Are the tool descriptions treated as sensitive data, or as public schema? Which one do you log when the agent picks the wrong tool?

iin1006h12

That's the exact framing behind hintgap - the annotation is the permission model, not documentation. If destructiveHint/readOnlyHint is missing, the agent reasons about blast radius from the name and description alone, and a vague description invites precisely the "guess the argument shape, server accepts a wider call" failure you describe.

On your two questions:

Sensitive or public schema? It's public schema - the description/annotation ships to any client that lists your tools, so it must never carry secrets, internal hostnames, or auth hints. But it's security-relevant public schema: it defines what the agent believes it's allowed to do, so it deserves review like an authz policy, not a changelog.

What to log when the agent picks the wrong tool? The decision context, not the payload: the tool name chosen, which annotations were present/absent at call time, and the shape of the arguments (keys/types, not values). That tells you whether the miss was a missing hint, an ambiguous description, or a genuine model error - which is exactly the signal you feed back into tightening the schema.

Search Hashnode

Your MCP tools never told the agent what they can do

Responses4