KNKartik N V J Kinkartiknvjk.hashnode.dev路14h ago 路 7 min readI picked a coding agent off a leaderboard. It flopped on our codebase.Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good about it. I pulled up the public benchmark scores, lined up the contenders, took the one at the top, 00
KNKartik N V J Kinkartiknvjk.hashnode.dev路4d ago 路 8 min readI built an LLM eval framework from scratch. Here is what I wish I had bought instead.One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed beautifully. I felt clever. Six months later that same framework was a mess. Three different judge models with00
KNKartik N V J Kinkartiknvjk.hashnode.dev路Jul 2 路 7 min readHow I catch the voice agent failures generic evals miss Voice agents break in ways that are specific to the job they do. A retail support agent gets the order number wrong. A clinical scribe drops a drug name. An insurance agent botches a coverage code. Th00
KNKartik N V J Kinkartiknvjk.hashnode.dev路Jun 25 路 6 min readHow I A/B test LLM prompts without fooling myselfI have shipped a prompt change on a number that meant nothing, and paid for it the next morning. The pattern is always the same. Someone makes a small tweak to the support agent's groundedness prompt.00
KNKartik N V J Kinkartiknvjk.hashnode.dev路Jun 16 路 9 min readHow I stopped my RAG CI gate from lying to meI have learned to distrust a fast green check on a RAG eval. The failure has a shape I now recognize. A PR lands late in the day. The RAG eval runs in 38 seconds and goes green. Twelve hours later the00