Raw accuracy fell 25 points at 50 tools, but a chance-corrected read tells a very different story. The request was simple: "Email Priya the Q3 numbers right now, subject 'Q3 Numbers'." The agent had a
Markdown supported · ⌘↵ to post
The chance-corrected framing is a useful correction to the raw-accuracy headline. With 20 scenarios and one trial each, a 55% vs 75% difference has a fairly wide confidence interval (a Wilson interval on n=20 is roughly +/-20 points), so repeating each pair a few times at temperature above 0 might show how much of the 25 to 50 cliff is noise. The find_contact result also made me wonder about two things: whether its description is broader than the others (a generic 'look up a person' tool can act as an attractor for any request that mentions a name), and whether menu order matters. Shuffling tool order between runs would separate a genuine attractor from position bias. Did you try either, or test a retrieval step that pre-filters the menu to the top-k tools?
Good call separating the three always-wrong scenarios from the tool-count effect. On your closing question, I would do the telemetry first. Most agents reach 50 tools by connecting a few MCP servers, not by design, so a per-tool failure count from your own logs usually points at one server, and the fix is often disconnecting it rather than adding a retrieval layer. It also matters what the sink tool can do. find_contact absorbing calls is a wasted turn. A generic update or send tool absorbing them is an action nobody asked for.
Cai
A custodial email, a stablecoin wallet, a credential vault, and an agent-ready API — all at one @cai.com address.
The find_contact absorption is a great signal -- one tool acting as a default sink across all menu sizes suggests the model falls back on a high-frequency embedding rather than re-scoring every option. This is exactly why protocol-level tool grouping (like MCP's per-server namespace separation) helps: it shrinks the effective decision surface by scoping the model to only the tools in the relevant domain. A flat 50-item menu forces discrimination across unrelated categories, while structured routing keeps each decision point small.