Good question. Our current offline relevance evaluation is human-led—so it’s actually the more expensive option 😄 We manually review query–result pairs against the original search intent and use those judgments as the quality benchmark for improving the Cloudsway Search API, rather than using an LLM judge as the final scorer.

And agreed on the retry cap—a grader alone doesn’t make the retrieval loop reliable. If you’d like, I’d be happy to give you access to test the API against your own evaluation setup. I’d especially value the failure cases you find.