jamesoconnor

Step-level agent evals exist now. Most teams still grade the finish line.

The thesis: an agent that fails at step 2 of 7 and an agent that fails at step 7 of 7 get the same score from an outcome eval, and they should not. The first one picked the wrong tool while holding th

Our agents reported success on tool calls that had already failed. Here's the pattern.

The tool-call number I published has expired. The claim it supported has not.

Function-calling robustness is a model selection problem masquerading as a prompt problem. I proved that with a percentage, which was the mistake. TL;DR. Two months ago I posted a number in a com

Parallel tool calls corrupted our shared state four times. Here's the pattern.

TL;DR. If your LLM agent uses tool calling, the model can return several tool calls in a single turn, and most agent loops execute those calls at the same time. For about a year I treated the bugs tha

The agent retried the tool call. The customer got charged twice.

An agent will re-issue a tool call for reasons the tool never sees, and if that tool moves money, sends mail, or creates a record, the retry runs the side effect again. Prompting the model to be caref