Production patterns for backend engineers who are tired of hello-world tutorials.
Open to Senior Full-Stack / Backend Engineer and Engineering Lead roles focused on cloud-native architecture, distributed systems, and AI integrations. Also open to freelance consulting on GCP migrations, FastAPI services, and infrastructure-as-code.
Good addition — secrets: inherit is the obvious convenience solution that creates a non-obvious governance problem. Once a shared workflow lives in a central repo, inheriting everything means the callee gets access to secrets it doesn't need, across every repo that calls it. Fine for a small team with a single org repo. Worth a deliberate decision rather than a default once you're managing secrets across multiple projects or teams.
"Treat the workflow interface like an API" is the right mental model and the post doesn't say it that clearly. Explicit inputs and outputs, documented permissions, no repository-specific assumptions baked in — those are the constraints that make a reusable workflow actually reusable rather than just shared. The hidden coupling problem is real and usually shows up the second time you try to use the workflow somewhere new.
Thanks — "assertion strategy problem" is the reframe that made the rest of the testing approach click for us too. The demo-to-production gap on AI features is real and mostly comes down to exactly this: the habits that work for deterministic code silently fail on probabilistic output. Glad it was useful.
This is the cleaner framing than what the post uses. Contract tests verify the application handles the response correctly — schema, error paths, retry bounds. Evaluation cases verify the model extracted the right thing. Conflating the two is what produces tests that are either flaky or meaningless. "Make each test responsible for a property that can actually be judged reliably" is the principle the post should have opened with.
You're right that schema validity and extraction correctness are different things, and the post conflates them. Asserting total_amount > 0 catches malformed output, not wrong output. The locale-aware comparison against a labelled expected value is the correct approach for testing extraction quality — it just belongs in the separate evaluation suite against known source documents, not the mocked unit tests. The mocked tests cover application behaviour; the evaluation suite covers whether the model extracted the right value. Both need to exist.
Solid edge case — finish_reason="length" is exactly the kind of failure that looks like success if you're only checking that the response came back. Checking it before any persistence call is the right place to gate it, and worth adding to the fixture suite alongside the malformed JSON case. The bounded retry matters too — a truncated response retried with the same prompt and same token limit will truncate again.
Good addition — temperature=0 plus a fixed seed is a useful middle ground, especially for integration tests where you want some real-API coverage but less noise. The caveat you flagged is the right one: seed behaviour isn't guaranteed across model versions, so it's a noise reducer not a flakiness eliminator. Still worth doing. I'd pair it with snapshotting the model version in the test so a version bump is a visible signal rather than a silent source of drift.
The normaliser for numbers and dates is exactly the right move — one fix, whole category gone. For free text: three things I actually assert on — required concept is present (loose, not keyword), length is within a sane range, and no model artefacts (leading "Sure!", markdown fences in plain-text fields). Beyond that I treat it as an evaluation problem, not a unit test problem. Weekly real API runs are where those quality regressions surface, not mocked tests.
Three that actually hurt us: Migrate and refactor separately. When something breaks mid-migration you won't know which change caused it. Move first, clean up after. Don't assume equivalent services behave the same. SQS and Pub/Sub are both "message queues." The delivery guarantees and retry behaviour are different enough to surprise you. Write a rollback plan per service, not just for the database. When a Cloud Run service had issues post-deploy, we were improvising. A one-page runbook per service removes the stress. Good rule of thumb: if a decision feels rushed mid-migration, it probably is.
