My agent said the refund went through. The ticket went to the one team that could not fix it.My refund agent told a customer the refund was processed. It wasn't. The payment tool had timed out, the harness retried twice, the third attempt came back with an empty body, and the output parser tu1MOct 13 路 5 min readkartiknvjk.hashnode.dev
I captured a hundred agent failures a week and fixed three. The bottleneck was curation.I captured a hundred agent failures a week and fixed three. The bottleneck was curation. My agent had the kind of observability setup I used to brag about. Every model call, every tool call, every retOct 8 路 5 min readkartiknvjk.hashnode.dev
Eval the Signature, Not the Program: How I Fixed My DSPy EvaluationLast month I shipped a DSPy RAG pipeline that looked done. I compiled it with MIPRO, the hold-out numbers were solid, CI was green. Two days into real traffic the quality complaints started. RetrievalOct 6 路 5 min readkartiknvjk.hashnode.dev
My CI evals were green. A regression still paged me at 3 AM.The eval that finally caught a bad regression for me was not the one in CI. CI had run twelve hours earlier, against a frozen test set, and passed. The regression was sitting in live production traffiSep 17 路 4 min readkartiknvjk.hashnode.dev
LlamaIndex makes RAG easy to build and hard to debug. Here is how I evaluate it.I built a RAG app on LlamaIndex in about four lines. Wire an index to a query engine, point it at my documents, ask a question, get an answer. The first hundred queries were great. I was impressed wit11KSep 2 路 8 min readkartiknvjk.hashnode.dev