Cost per task is the metric I wish more teams tracked from day one. One thing that helped us: tagging every span with the original user goal ID, so when an agent loops through five retries you can see the whole chain priced against one outcome instead of five 'successful' calls. It also makes it easy to spot tasks where the agent technically finished but burned 10x the median budget, which is usually where the prompt or tool description needs fixing.
Yunus Emre Vurgun
Cost per task is the metric I wish more teams tracked from day one. One thing that helped us: tagging every span with the original user goal ID, so when an agent loops through five retries you can see the whole chain priced against one outcome instead of five 'successful' calls. It also makes it easy to spot tasks where the agent technically finished but burned 10x the median budget, which is usually where the prompt or tool description needs fixing.