Comment by Mikhail Golikov on "From Chatbot to AI Agent: How Conversational Automation Is Evolving"

The "testing and debugging tools" line in your AI agent platform checklist is the part that hits closest to real production pain. We built pytest-conversational for the same reason: when an AI agent handles 8 turns of multi-step support flow, manual QA can't reliably catch when turn 5 stops following the deterministic part of the workflow because the LLM reasoning step started leaking past the controlled boundary.

The approach we landed on: keep the assertions deterministic (rule-based matchers on turn N response shape, role-based permissions, expected tool calls), and let only the agent reasoning be probabilistic. So tests can express things like "if user says X at turn 3, system MUST call tool Y with parameter Z extracted from turn 1, and MUST NOT skip the human handoff trigger at turn 5". Zero LLM in the test side - fully reproducible, runnable in CI without token costs.

What surprised us: the more flexible the agent (Hexabot pattern, structured workflow + AI reasoning), the more important determinism on the test side becomes. If both the agent AND the test suite use AI to evaluate output, you have two non-deterministic systems judging each other and the failure modes compound silently.

Question: how do you handle workflow versioning for tests? If a tool changes signature between versions of the workflow, do test fixtures auto-detect mismatch or do you rely on integration tests catching the drift after deploy?

This is exactly the right framing. I strongly agree with keeping the test side deterministic. Once the agent and the evaluator are both probabilistic, debugging becomes much harder because you no longer know whether the failure came from the workflow, the model, or the test judge.

In Hexabot, we usually think about this in two layers:

First, the action/tool contract should be tested as regular code. When developing a custom action, we expect the action schema, parameters, output shape, error handling, and edge cases to be covered with unit tests. That’s the first place where a tool signature change should be caught before it affects workflows: docs.hexabot.ai/developer-guide/develop-custom-ac…

Second, workflow tests should be tied to workflow versions. Hexabot separates drafts from published versions, so the version that runs in production is not just “whatever is currently being edited.” This makes it possible to reason about tests against a specific workflow artifact/version: docs.hexabot.ai/workflow-editor/versions-drafts-a…

For workflow-level testing, the current approach is to export the workflow and generate a test file from it. I would treat those generated tests as version-coupled fixtures: if the workflow changes, or if an action contract changes, the test should either fail clearly or be intentionally regenerated/reviewed. I would not want signature drift to be discovered only after deploy through integration tests.

So practically: action unit tests catch tool-level contract drift, exported workflow tests catch workflow-level behavior drift, and published workflow versions give us a stable reference point for what is actually running.

Search Hashnode