Software development, technical writing, consulting, and speaking on DevOps, platform engineering, and applied AI.
Great addition, and a sneaky one, because the pool looks healthy while one slow endpoint holds a backend hostage. Agree on setting idle_in_transaction_session_timeout as the backstop. The real fix is usually moving the API call or queue publish outside the transaction, but the timeout catches the paths nobody remembers. Your pg_stat_activity query is the first thing I'd run too.
Both fair, and the first one is exactly where I ended up. Labels now point at the answer text instead of chunk IDs, so a chunking change doesn't quietly move the goalposts. Agree on the judge too. A small hand-labeled set to check it against first is cheap, and a judge that leans SUPPORTED is worse than no judge, since it looks like it's working. Might fold both into the post.
Same week lost here, so I feel this one. I stopped storing chunk IDs for exactly that reason. Each eval case now stores the source doc plus the exact answer text, and a retrieved chunk counts as a hit if it contains that text. Change the chunking and the eval set still works, no relabeling. Nice write-up on the CI gate, thanks for sharing it.
