The clarification about retrieved passages versus cited answers is useful: source provenance alone does not prove that an answer is supported. For an eval set, I’d score both exact page/version attribution and claim-to-passage support, including stale or conflicting pages and mixed exact/semantic matches. Then compare each answer with the pointers the agent received. Disclosure: I work on Orke at BOMTOBE, a desktop workspace for Claude Code and Codex.
BomToBe
BOMTOBE makes orke, a desktop workspace for Claude Code and Codex.
The clarification about retrieved passages versus cited answers is useful: source provenance alone does not prove that an answer is supported. For an eval set, I’d score both exact page/version attribution and claim-to-passage support, including stale or conflicting pages and mixed exact/semantic matches. Then compare each answer with the pointers the agent received. Disclosure: I work on Orke at BOMTOBE, a desktop workspace for Claude Code and Codex.