A line such as "checkout failed: " + error feels useful on a laptop. In production it cannot reliably answer which release introduced the failure, which dependency is involved, how the request crossed services, or whether real orders were affected. String concatenation can also send a token, email address, or complete request body into a long-lived logging platform.

Structured logging is not a sentence wrapped in JSON. It is a versioned interface for production events, with security and cost boundaries.

Record an event before writing prose

Give each event class a stable, low-cardinality name such as checkout.payment_failed. Do not embed an order identifier in that name. Fields describe the occurrence; the body remains a short human-readable summary:

{
  "event.name": "checkout.payment_failed",
  "event.version": 1,
  "severity_text": "ERROR",
  "service.name": "checkout-api",
  "service.version": "2026.10.13",
  "deployment.environment": "production",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "payment.provider": "example-pay",
  "error.type": "ProviderTimeout",
  "retryable": true,
  "body": "Payment provider timed out"
}

The OpenTelemetry log data model distinguishes the event timestamp from the observed timestamp, numeric and textual severity, body, resource, instrumentation scope, attributes, and optional trace and span identifiers. Service name, release, and deployment environment are Resource properties because they remain fixed for the process. Provider, result, and retryability are event Attributes because they vary per occurrence. Mixing the two increases repetition and weakens query semantics.

Version the event schema. Increment the version when removing a field or changing a unit or enum meaning, and give dashboards and alerts a migration window. Logs are queried, archived, and sometimes audited. “Only debugging output” is not permission to change their meaning silently.

Severity describes the required response

Use ERROR when the current operation failed and requires investigation or an explicit fallback. Use WARN when work completed but approached a risk boundary, INFO for meaningful lifecycle and business outcomes, and DEBUG for temporary diagnosis. Logging every correctly rejected user input as an error trains responders to ignore the red signal.

Record exceptions through structured exception attributes: type, a sanitized message, and stack trace. Do not duplicate the entire stack in the body and several fields. Log once at the boundary that handles the exception, adding decisions such as retry, fallback, and final outcome. An intermediate layer that learned nothing new should propagate the error rather than produce another copy.

Trace context correlates work; it is not identity

On an incoming HTTP request, parse and validate W3C traceparent, create the current span, and propagate updated context downstream. Writing the active trace and span identifiers into logs allows an operator to move between events and a distributed trace. Asynchronous messages need explicit propagation or links; thread-local context will not reliably survive a queue boundary.

A trace ID is not a user ID, idempotency key, or authorization credential. W3C Trace Context prohibits personally identifiable or sensitive information in traceparent and tracestate. Treat public context as untrusted input: validate length and syntax, bound vendor state, and restart a trace at a security boundary when policy requires it. The incoming sampled flag is a hint, not permission for an external caller to control internal cost or retention.

Redact at the logging API boundary

Never record passwords, access or refresh tokens, session cookies, private keys, complete payment data, or unreviewed request and response bodies. A small allowlist of approved fields is more dependable than an expanding denylist of sensitive key names. When user correlation is necessary, use an internal irreversible reference with an explicit retention period. A plain hash may still be enumerable and linkable; it is not automatically anonymous.

Redaction must occur before an event enters a queue or exporter. Platform-side masking is too late if the secret has already crossed a memory buffer, standard output, or sidecar. Apply the same policy to exception messages and stacks, because URLs, database errors, and third-party SDKs frequently include credentials in text.

External input can also contain newlines, delimiters, or terminal control characters. Let a serializer escape it correctly, and sanitize CR, LF, and other controls in text-compatible sinks to prevent log injection. Never assemble JSON manually.

Set budgets for cardinality and volume

Event name, service, environment, outcome, and error type make useful common index fields. Order IDs, user references, raw URL queries, and stack traces have high cardinality; keep them as on-demand attributes or in controlled storage. Dynamic values in field names or metric labels rapidly increase index cost and degrade queries.

Limit record size, attribute count, string length, and nesting depth. Large bodies, binary data, and complete database rows belong in more restricted diagnostic artifacts; logs can contain a reference with its own expiry policy. If truncation occurs, add log.truncated=true and the original length so responders do not mistake partial evidence for complete content.

Sample by event value. Deterministic sampling can reduce repetitive successful health checks, but errors, deployment events, and security audit records generally need stronger retention. Audit logging should have separate access control and reliable delivery. It is not merely another verbosity level in the application stream.

Make failure of the logging path observable

An asynchronous logging pipeline needs a bounded queue. Decide whether saturation blocks requests, drops low-priority events, or writes to a local emergency sink, and expose counters for that decision. An unbounded buffer converts an observability outage into an application memory outage. Silent dropping removes evidence precisely during an incident.

Measure export delay, dropped records, serialization failures, redaction matches, export retries, and backend rejection. The logging system itself needs metrics and alerts, but an exporter failure must not recursively report through the same exporter.

Test the log contract

Schema-test critical events: required fields exist, types and enums remain stable, timestamps parse, and severity follows policy. Security tests inject tokens, cookies, email addresses, newlines, and oversized strings. Assert that final sink output contains no original secret and still parses. Propagation tests should cross two services and a queue without losing trace correlation.

Load tests should stop or slow the collector and verify queue limits, drop priority, and application latency. Finally, validate queries with real incident questions: can an operator compare failure rate by release, move from an alert to a trace, and find the final error within minutes without reading a user secret?

Before release, confirm stable event names and versions; separation of Resource fields and event Attributes; actionable severity rules; one exception record at the decision boundary; validated Trace Context without identity data; redaction before export; no dynamic field names; explicit size, cardinality, sampling, and retention budgets; and tested, observable overload behavior.

Good logging is not maximum logging. Every retained event should answer a production question. A stable schema makes machine queries dependable, trace context connects events into a causal path, and redaction plus budgets prevent visibility from creating a new security or cost incident.