Before putting them into production is worth more time, because the boundary that matters here is what the agent is allowed to touch when it is wrong. Write down the permissions before the evaluation set, every time, because you cannot measure an agent honestly until you have decided what it is allowed to reach when it is wrong. Tell me how you keep the evaluation set honest once the agent starts shaping the work it's being measured on.