Jjamesoconnorinjamesoconnor.hashnode.dev·4d ago · 8 min readSix was the number I typed. My simulated user stopped there for six weeks. We built a multi-turn harness for our support agent after a bug that a single-turn eval could not have caught. A customer asked about one account, got a good answer, then switched to a second account 00
Jjamesoconnorinjamesoconnor.hashnode.dev·4d ago · 5 min readStructured output broke on us three times. The third time taught us what "operator-ready" means.Last quarter we shipped a contract-extraction agent to an enterprise legal team. Schema validation passing at 97%. Human reviewers satisfied with the output quality in testing. Rollout went smoothly. 00
Jjamesoconnorinjamesoconnor.hashnode.dev·Aug 14 · 12 min readNobody counts guardrails the same way twice, so "we added guardrails" tells you nothingTL;DR: Counting what six guardrail tools ship is harder than it sounds, and I got one of them wrong on the first pass. The answers are 27, 0, 12 or 31, 14, 12 plus 15 plus 4, and one that cannot be co00
Jjamesoconnorinjamesoconnor.hashnode.dev·Aug 12 · 10 min readNearly half our agent's tool-call failures were our assertion, not our agent We had a number on a dashboard and it would not move. Twenty-two percent of the tool calls our agent made were scored as failures by our eval suite. We spent most of a quarter on it: better tool descr00
Jjamesoconnorinjamesoconnor.hashnode.dev·Aug 7 · 8 min readEvery trace-capture tool gives you one assertion slot. Your agent failed on step three.Every observability tool I use has a button that turns a production trace into a dataset row. I have clicked it a lot, on the assumption that this was the loop closing: agent fails, capture the failur00