One edge case worth covering is config drift across replicas during a rolling deployment. Imagine a payment service where the new release introduces a required settlement_mode, but one replica still runs with an older config schema while another picks up the new settings. Both processes might start successfully, yet handle the same settlement differently.
I'd validate the config against the exact schema expected by each release and expose a config version or fingerprint in the startup logs and health checks. For runtime reloads, I'd validate the complete candidate config first, then swap it in atomically, keeping the last known-good config if validation fails. Kubernetes uses this general pattern for manifest-based admission control, which is a useful reference here.
Startup validation catches bad initial state, but it doesn't solve every config problem. Schema compatibility during deployments and safe reloads are where things can get interesting.
One edge case worth covering is config drift across replicas during a rolling deployment. Imagine a payment service where the new release introduces a required
settlement_mode, but one replica still runs with an older config schema while another picks up the new settings. Both processes might start successfully, yet handle the same settlement differently.I'd validate the config against the exact schema expected by each release and expose a config version or fingerprint in the startup logs and health checks. For runtime reloads, I'd validate the complete candidate config first, then swap it in atomically, keeping the last known-good config if validation fails. Kubernetes uses this general pattern for manifest-based admission control, which is a useful reference here.
Startup validation catches bad initial state, but it doesn't solve every config problem. Schema compatibility during deployments and safe reloads are where things can get interesting.