The part that stands out is treating credential rotation as a state transition rather than a secret update. Changing the database password, updating consumers, and then verifying the running system are three different states, and skipping the last one is where “successful” rotations turn into delayed outages.
One thing I’d add is making the verification itself part of the rotation procedure: test an actual read/write path from each dependent service, not just container health. That catches cases where the container restarted successfully but is still using stale configuration, a different secret source, or a connection pool that never re-authenticated. The real success criterion is restored application behavior, not just a changed password.
Awesome! That's a definite gap in what I described. Container health and application health aren't the same thing — a container can report healthy while a connection pool inside it is still holding a stale secret from before the restart, especially if the pool doesn't force re-authentication on reconnect.
SELECT COUNT(*) told me the data survived. It didn't tell me every service was actually serving traffic on the new credential versus a cached one. Those are different claims, and I was only checking the first.
Folding this in going forward: an actual read/write check per dependent service, not just a row count or a container status, before calling a rotation done. Thanks Mateo!
Mateo Ruiz
The part that stands out is treating credential rotation as a state transition rather than a secret update. Changing the database password, updating consumers, and then verifying the running system are three different states, and skipping the last one is where “successful” rotations turn into delayed outages.
One thing I’d add is making the verification itself part of the rotation procedure: test an actual read/write path from each dependent service, not just container health. That catches cases where the container restarted successfully but is still using stale configuration, a different secret source, or a connection pool that never re-authenticated. The real success criterion is restored application behavior, not just a changed password.