Yup, I have now measured it inside one language. I rewrote the 20 English golden questions in a customer's words. "Bank" for PSP, "instant transfer" for SCT Inst, none of the section's own terms. The original wording puts the answer page in the top 5 for 17 of 21 questions, when the everyday wording does it for 14. One example: "available on weekends and public holidays" finds the page at rank 2 vs. "Can I send an instant transfer on a Sunday?" finds it at rank 46. The page says "Calendar Days". So a golden set in the rulebook's own words does look healthier than real questions. Here the gap is 3 of 21, before any language change. On translating the query first: I measured it in this article. It brought Russian to within one question of English and did nothing for Spanish. Two questions already sat at rank 4 and 5 in English, so any rewording pushes them out. Inspectable, agreed. It is also a paraphrase, and the fragile questions pay for it. The phrasing numbers go out as the next piece.
@skucherenko
About
Software architect. 10 years of production systems. Now building AI systems with citations, evals, and a written reason for every decision.
Available for
Nothing here yet.