Calling server.close() after receiving SIGTERM does not complete a graceful shutdown. A load balancer may still send requests, an HTTP/2 connection can create more streams, a consumer may have just leased a message, and a database transaction may not have committed. Waiting only for an “active requests” counter to reach zero lets one long-lived connection block a deployment forever. Exiting immediately leaves clients with ambiguous results and potentially duplicated side effects.

Graceful shutdown is a bounded distributed protocol: stop taking new work, finish accepted work within a budget, preserve recoverable state, and guarantee that the process eventually exits.

Define a state machine first

Make the lifecycle explicit:

SERVING -> QUIESCING -> DRAINING -> STOPPED
  • SERVING accepts traffic and background work.
  • QUIESCING leaves service discovery and stops claiming jobs.
  • DRAINING handles only requests, RPCs, and messages already promised.
  • STOPPED closes resources and exits.

The shutdown entry point must be idempotent. A repeated signal, administrative endpoint, and orchestrator callback can all trigger it, but state only moves forward. A second termination signal can intentionally mean “exit immediately,” giving operators a way to handle a stuck instance. Document that behavior in the runbook.

Stop new work before waiting for old work

First fail readiness and stop polling queues. Then close listeners or invoke the framework’s graceful-stop API so it rejects new connections and RPCs. Service discovery, proxies, and the process do not update atomically. There is a propagation window between changing readiness and traffic actually stopping, so the application must reject new work during that window rather than relying on a fixed sleep.

In Kubernetes, a terminating Pod’s EndpointSlice endpoint has ready set to false and is marked as terminating. The node also begins the container termination flow; do not assume every load balancer observes the endpoint update first. If a preStop hook exists, it runs before the TERM signal, but the terminationGracePeriodSeconds countdown has already started. A hook that consumes 25 seconds leaves only five seconds of a 30-second budget for the application.

Use preStop for short protocol actions, not a long sleep as a correctness mechanism. The application still needs to handle termination, change its health state, and drain work.

Connection shutdown has protocol semantics

Stopping new TCP accepts does not settle existing connections. An HTTP/1.1 keep-alive client may reuse a connection, WebSocket or SSE sessions can last for hours, and HTTP/2 multiplexes many streams over one connection.

HTTP/2 GOAWAY lets an endpoint stop accepting new streams while finishing established ones. It carries the last stream identifier that might have been processed, so a client knows that higher-numbered streams can be retried on a new connection. Closing without GOAWAY can leave a client unable to tell whether a POST began producing side effects. Use the server framework’s protocol-aware graceful shutdown instead of emulating it by closing a socket.

gRPC graceful shutdown similarly rejects new RPCs and allows in-flight RPCs to complete. Its official guidance also requires a deadline followed by forceful shutdown. Clients need healthy alternative endpoints, and retries still require deadlines, idempotency, and a retry budget.

Long-lived sessions need their own product contract: send a close frame or reconnect hint, allow migration time, then disconnect at the deadline. If seamless migration matters, session state cannot exist only in the memory of the process that is leaving.

Split the total grace period into budgets

If the orchestrator grants 30 seconds, do not give all 30 to request draining. Allocate it deliberately:

0s      fail readiness; stop claiming jobs and accepting connections
0-22s   drain requests, RPCs, and already claimed messages
22-27s  cancel remaining work; flush essential telemetry and checkpoints
27-30s  close pools and preserve a margin for forced termination
30s     orchestrator hard stop

The internal deadline must precede the external hard limit and propagate to child calls. Once shutdown starts, do not begin retries, batches, or callbacks that exceed the remaining budget. Request handlers must observe cancellation; canceling a framework context does nothing if business code ignores it.

Choose the grace period from real latency percentiles and the longest legitimate operation. If a valid request sometimes needs two minutes, a 30-second termination window cannot both complete it and deploy on time. Shorten the request, convert it into a recoverable job, or change the deployment budget.

Define the commitment point for messages and side effects

On shutdown, a consumer stops polling before it drains claimed messages. It acknowledges a message or commits an offset only after the business side effect and required outbox record commit. If the deadline arrives, release the lease or let visibility timeout restore the message; never acknowledge early merely to empty the local queue.

The process can receive SIGKILL, crash, or lose power, so every shutdown hook can be skipped. Correctness cannot depend on one final in-memory checkpoint. Message processing needs idempotency keys, leases need expiry, and critical progress needs continuous durable storage. Graceful shutdown reduces interruption; it is not a crash-consistency mechanism.

Close the database pool after requests and consumers drain. Closing it early breaks transactions that could have completed, while waiting forever consumes the entire budget. Release resources in reverse dependency order, with every step bounded by the same remaining deadline.

Make shutdown observable and testable

Record state-transition times, received signals, active work at start, newly rejected work, completed and canceled counts, oldest-work age, final exit reason, and whether force was required. Centralized telemetry must still answer what happened to the last requests after the instance disappears. Give telemetry flush a small bounded budget; it must never prevent exit.

Test idle instances, persistent keep-alive traffic, concurrent HTTP/2 streams, long RPCs, slow database transactions, active queue consumption, a second signal, duplicate shutdown triggers, a stuck dependency, and grace-period expiry. An end-to-end test should roll a deployment under continuous traffic and verify client error rate, duplicate effects, message redelivery, and maximum termination time—not merely unit-test the signal handler.

Before release, confirm that new traffic is reliably rejected; the protocol layer emits the proper shutdown signal; every work type has a commitment point and recovery path; the internal deadline is shorter than the orchestrator grace period; resources close in bounded reverse order; the force path is tested; and correctness never assumes that a shutdown hook will run.

Graceful shutdown does not mean waiting forever for every operation. It means reducing ambiguity inside an explicit time window. Once states, commitment points, deadlines, and recovery mechanisms are written down, routine instance replacement stops becoming a source of random user-visible failures.