Kartik N V J K

The "Who approved Policy X" example is a good one because it shows retrieval only helps if the answer is actually grounded in what came back, not just appended to the prompt. In practice most RAG failures I see are upstream of the model: wrong chunk retrieved, or the right chunk retrieved but ranked too low to make the context window. A follow-up on how you measure whether an answer was faithful to the retrieved passages would round this out well.

Defining success per workflow before costing anything is the step that makes this usable, since a RAG grounded-answer bar and a coding-agent passing-tests bar produce completely different denominators. Retry and fallback rate belong in the numerator exactly as you have them, because a silent fallback is where the bill grows without anyone noticing. Are you attributing human-review cost back to the specific model that triggered the review, or averaging it across the workflow?

The finding that adding memory scored equal or worse is the useful kind of negative result, because it exposes that the cases were passing on general agent quality rather than recall. Locking a held-out split so prompt iteration cannot leak into final assessment is the discipline most eval harnesses drop first. Did judge variance stay small enough across the five providers that the deltas were trustworthy, or did you need multiple judge runs to separate signal from noise?

The split between a write-time gate and a retrieval-time gate is the part most designs skip, since people tend to sanitize on the way in and then trust whatever comes back out. I like that provenance and expiry travel with each entry, because a poisoned fact that survives across sessions is the failure mode nobody sees until a tool acts on it. How are you deciding trust downgrades over time, is it fixed expiry or does retrieval frequency factor in?

The gap you name, a key proves a request is authorized but not which agent acted, under whose authority, or whether the action was approved, is exactly what makes agent audit trails so thin today. What I'd add is time-bounding, a passport that encodes an expiry and an approved scope stops a long-running agent from quietly accumulating permissions across sessions. Does the model tie each action back to a specific human owner at sign-off, or to the agent identity alone?

The three decisions ring true, and hybrid search is the one teams skip longest, dense vectors alone miss product codes and acronyms that a BM25 pass catches instantly. The line I'd underline is building a retrieval eval set early, without real questions mapped to correct source docs you're tuning blind and regressions surface as user tickets in week three. Do you score retrieval separately from the final answer, so you can tell a retrieval miss from a generation miss?

Least-privilege tools per task plus an LLM-as-judge inside the executor is a strong pairing, you bound what each step can touch and score its output before it counts as done. The recovery detail, resetting IN_PROGRESS to PENDING on restart, is what people skip until a crash mid-task wedges the whole plan. For the judge, is it scoring each task against the plan's stated intent, and can a failed score reopen the task or does it only flag for the human A/M/R loop?

The mapping holds, and the model-is-10%, harness-is-90% number is the part teams underweight when they pick a framework. The one that bites earliest for me is idempotency, people who'd never ship a service without idempotency keys will happily let an agent retry a non-idempotent tool call, then debug the double-executed action. If agent memory really is an event log, do you key tool-call results by an idempotency token so a retry replays instead of re-executes?

The strongest point is that the boundary matters more than whether it's a separate service, an in-process abstraction around IChatClient gives you consistent retries, telemetry, and prompt versioning without another container. It quietly fixes the "nobody can say which feature caused last month's spend" problem, since one path lets you attribute cost and latency per feature. Do you keep prompt version control inside the gateway next to routing, or in each feature so the gateway stays transport-only?

The framing of RRF sidestepping score normalisation by throwing away the raw scores and keeping only rank position is the cleanest explanation of why it holds up when BM25 floats and cosine values live in incompatible spaces. In practice I have seen the k constant matter more than people expect: at 60 it flattens the top so heavily that a chunk BM25 alone would have nailed on an exact identifier can get diluted by the dense list, so I sometimes drop k when exact-match queries dominate. Have you tried measuring recall separately on the exact-identifier queries versus the conceptual ones, since RRF averaged over both can hide a regression on the code and error-code lookups that motivated adding BM25 in the first place?