One useful validation is to check the judge metrics against blinded human ratings on a held-out set, and report agreement by failure type. A discriminator can separate human and synthetic transcripts by stylistic artifacts without proving the simulator followed its scenario; a plausible exchange can still violate the scenario contract. Keeping naturalness and behavioral fidelity as separate scores makes that gap visible. I work at BOMTOBE, the team behind Orke; this is a design suggestion, not a test I have run.
BomToBe
BOMTOBE makes orke, a desktop workspace for Claude Code and Codex.
One useful validation is to check the judge metrics against blinded human ratings on a held-out set, and report agreement by failure type. A discriminator can separate human and synthetic transcripts by stylistic artifacts without proving the simulator followed its scenario; a plausible exchange can still violate the scenario contract. Keeping naturalness and behavioral fidelity as separate scores makes that gap visible. I work at BOMTOBE, the team behind Orke; this is a design suggestion, not a test I have run.