Building and breaking things in the cloud
AWS cloud architect and AI enthusiast based in Morgantown, WV. I write about building on AWS, exploring AI agents and automation, and the occasional deep dive into cloud security. Follow along as I share what I'm learning, building, and breaking in the cloud
Markdown supported · ⌘↵ to post
Mateo Ruiz
The interesting part here is that the harness makes agent infrastructure portable, but the live-testing section shows why portability alone isn't enough. Once context handling, reasoning replay, caching, and model-specific behavior become runtime assumptions, those assumptions need to be characterized rather than buried in configuration.
I especially like keeping the hook even after the documented failure couldn't be reproduced. That’s a good production pattern: distinguish “currently reproducible” from “still worth guarding against,” and encode the expected behavior in a characterization test. Otherwise a future Bedrock/model revision can silently invalidate an assumption that nobody remembers was there.
For long-running agents, I’d treat those characterization tests almost like compatibility contracts between the harness and model provider. The model string may be swappable, but the behavioral contract around context, tool calls, reasoning state, and session replay is what actually determines whether the agent remains reliable.