Good question. Our current offline relevance evaluation is human-led—so it’s actually the more expensive option 😄 We manually review query–result pairs against the original search intent and use those judgments as the quality benchmark for improving the Cloudsway Search API, rather than using an LLM judge as the final scorer.
And agreed on the retry cap—a grader alone doesn’t make the retrieval loop reliable. If you’d like, I’d be happy to give you access to test the API against your own evaluation setup. I’d especially value the failure cases you find.
Provenance travelling with the content from the moment it enters the pipeline is the part most teams skip, and it breaks citations later. I grade retrieval before generation too, but the retry cap matters as much as the grader. I wrote about keeping that RAG gate honest here: kartiknvjk.hashnode.dev/how-i-stopped-my-rag-ci-g…. Are you scoring relevance with an LLM judge or something cheaper?