“Generate a new value, update the Secret, restart the service” sounds like a rotation. In production it can break every database pool, reject instances that have not restarted, and invalidate tokens that are still legitimately in flight. Producers, consumers, verifiers, and caches do not switch at one instant. Overwriting one value merely disguises a distributed migration as a configuration edit.

A zero-downtime rotation needs a reversible protocol: old and new versions coexist briefly, new writes use only the new version, reads or verification accept the old version for a bounded window, and revocation happens only after evidence shows that old use has ended.

Inventory secrets and give versions identities

Record each secret’s purpose, owner, issuer, consumers, environment, privilege scope, creation and expiry, rotation procedure, and emergency contact. A database password, third-party API key, HMAC secret, TLS private key, and data-encryption key cannot share one playbook. They have different verifiers, revocation capabilities, and historical-read requirements.

Consumers should reference a logical name plus a concrete version, such as payments/db-password version v42, instead of one opaque “current” value that can change silently. Log the version ID and operation result, never the secret, connection string, or reversible digest. Metadata must answer who created and read a version, when it became active, and when it was revoked.

Use expand, migrate, and contract

A planned rotation has four stages:

  1. Prepare a least-privilege version and test its target identity, format, and rollback path without serving production traffic.
  2. Expand the authenticator or verifier to accept both versions and authorize every intended consumer to retrieve the new one.
  3. Migrate producers so only the new version signs, encrypts, or creates connections. Keep the old version only for historical verification and draining existing clients.
  4. Contract after the longest token, cache, job, and connection lifetime passes. Prove old usage is zero, revoke it, and verify rejection.

Every step needs an idempotent state transition. A rotation function must not depend on a variable left in memory by an earlier invocation. Rollback changes which version is active; it must never put an old value back into source control or an ordinary configuration file.

Database credentials require pool migration

Changing the password of one database user makes updated and non-updated instances race. Existing connections might continue briefly, but any reconnect using the old password fails. A safer pattern creates a second principal, copies and verifies its least privileges, deploys the new credential, routes new connections into a new pool, drains the old pool, and then revokes the old principal.

For hot reload, parse the complete configuration and test a new connection before atomically swapping the active client reference. Keep the old client only for requests already in progress, with a drain deadline. If validation fails, retain the old version and alert rather than distributing a broken value to every instance. Database grants and secret publication also need an auditable order.

Migrate signing and verification separately

A signer should use one active private key for new signatures, while verifiers retain multiple public keys during rollover. A token or message carries a key ID so the verifier can select the right key. RFC 7517 defines kid for matching a particular key in a JWK Set, including during rollover.

Publish the new public key and prove every verifier can see it before switching the signer to the new private key. Remove the old public key only after the maximum old-token lifetime, clock skew, and queue delay have passed. Accepting multiple key IDs does not mean accepting arbitrary algorithms: algorithm, issuer, audience, and intended use remain fixed checks.

A webhook that uses a shared HMAC secret similarly needs a version field or an ordered verification strategy. If the remote provider supports only one active value, true dual-version migration is impossible. Follow its cutover protocol and make side effects idempotent.

Encryption-key rotation also affects old data

Store the key version with every ciphertext. After rollover, new writes use the active key and readers temporarily keep the old read set. A background job then rewraps data-encryption keys or re-encrypts historical records. That job must be resumable and rate-limited, and it must report how much data remains under each version.

Destroy old decryption material only after a coverage scan reaches zero and backup and archive policy accounts for old ciphertext. NIST key-management guidance distinguishes active use, archival, recovery, and destruction. “No longer encrypting” does not mean “safe to lose decryption immediately.” Envelope encryption can separate rotating a key-encryption key from rewriting all business data.

Distribution does not prove consumption

A running Kubernetes container does not observe a Secret environment-variable update; it needs a restart. A Secret mounted as a volume updates with eventual consistency, but a subPath mount does not receive automated updates. The application still needs to watch or periodically reread the source and atomically swap a complete in-memory configuration. A changed file is not proof that the credential works, so validate format, scope, and a real authentication attempt.

Short-lived dynamic credentials reduce the exposure window, but consumers must renew before lease expiry with jitter so every instance does not refresh together. Coalesce refreshes per credential. On failure, continue only within the remaining valid lease and alert early; never extend an expired credential indefinitely.

Revocation needs evidence

Dashboards should report issuance, reads, authentication success and failure, last use, renewal errors, and old-version hits by version—without exposing the secret. Before revocation, automation checks every registered consumer’s version, active connections, maximum token lifetime, queued work, and historical-ciphertext coverage. Afterward, actively prove that the old credential is rejected and the new credential still succeeds.

Planned rotation permits bounded overlap. Suspected compromise is incident response and may require immediate revocation with some failed requests. The emergency plan must reduce privilege, disable one version, identify affected callers, and distribute safe replacements rather than treating an incident as a faster scheduled script.

Tests should cover an invalid new version, a partially refreshed fleet, pool-drain timeout, verifiers caching an old JWK Set, duplicate and reordered events, delayed Secret volume updates, lease expiry, historical ciphertext using multiple versions, and rollback halfway through. Also rehearse whether errors after revocation are actionable without leaking sensitive material.

Rotation is not complete when a new value exists. It is complete when every intended path uses the new version, residual dependency on the old version is measurable, and revocation has been verified. Treating rotation as a stateful migration shortens credential exposure without turning a security control into an outage.