A cache is often abstracted as get(key) and set(key, value, ttl), yet production failures rarely come from those two calls. They come from serving one tenant’s data to another, showing an old price after an update, expiring every hot key together, or letting a browser, CDN, and Redis use incompatible definitions of “fresh.”

A reliable cache answers three questions first: which requests represent the same content, how stale that content may become, and how every layer converges after a change. Hit rate is an outcome, not the contract.

A key must identify the complete representation

The same URL can vary by tenant, authorization, language, device capability, content encoding, or experiment cohort. HTTP caches use Vary to name request fields that select a representation. An application cache needs the equivalent: one centrally maintained key builder instead of strings assembled at call sites.

product:v3:{tenant_id}:{product_id}:{currency}:{locale}

Version the schema and define normalization, casing, and missing values. Never put raw tokens, email addresses, or other secrets into keys and logs. More importantly, “signed in” is not a substitute for authorization context. If permissions change the response, use a private cache, include a safe security-principal dimension, or do not cache it. A shared HTTP cache can reuse a response to a request containing Authorization only when the response explicitly permits it.

Vary: * prevents a stored response from matching. Adding a high-cardinality header to Vary can also create a fragmented cache with almost no reuse. Every dimension needs a product reason and a test case.

Freshness and storage are different decisions

Cache-Control: no-store means do not store the response. no-cache does not mean “do not cache”; it requires successful origin validation before each reuse. private prevents a shared cache from storing a response while still allowing a private browser cache. max-age sets freshness, and s-maxage can set a separate lifetime for shared caches such as CDNs.

Cache-Control: public, max-age=60, s-maxage=300,
  stale-while-revalidate=30, stale-if-error=600

Here, a browser can use the response as fresh for 60 seconds and a shared cache for 300. The cache may then serve stale for 30 seconds while validating in the background, and may serve stale for a bounded period when the origin fails. These numbers express business tolerance; they should not be inferred from a latency target. Account balances, revoked permissions, and inventory promises usually need stricter stale limits than an article index.

Use immutable only for content-addressed or versioned URLs. Declaring /app.js immutable while replacing it in place leaves clients unable to discover the change until expiry. Prefer hashed filenames and long lifetimes for static assets, with short freshness and validators for HTML or entry manifests.

Validators save transfer without repeating all the work

An origin can return ETag or Last-Modified. After freshness expires, a cache sends If-None-Match or If-Modified-Since. If the representation has not changed, the server returns 304 Not Modified and refreshes stored metadata without transferring the body.

An ETag must identify the actual representation. Different JSON fields, languages, or content encodings cannot accidentally share one strong ETag; compression negotiation also needs Vary: Accept-Encoding. A weak ETag is useful when representations are semantically equivalent but not byte-for-byte identical. It cannot stand in for byte identity where range requests or write preconditions require it. Validation should be cheaper than running the full query and render path again, or it saves bandwidth without reducing origin work.

Hot expiry needs request coalescing

When ten thousand requests find one hot entry expired, they must not produce ten thousand origin calls. Each process or cache node should single-flight work for that key: one request rebuilds while others briefly wait or receive stale content when policy allows it. A lock needs a timeout and ownership check so a failed builder cannot leave a permanent lock.

Add controlled jitter to TTLs so large key sets do not expire on the same second. Background refresh should be triggered by real traffic or a bounded scheduler, with concurrency limits and propagated deadlines. Unlimited prewarming merely moves a stampede from request threads into a job queue. Negative caching needs the same care: a confirmed 404 can have a short lifetime, but a timeout or 500 must not become a durable “missing,” and authorization failures must never be shared across principals.

Invalidation has to cross every layer

One request can pass through a browser, CDN, process memory, and Redis. Deleting only the Redis key leaves old browser content intact; purging only the CDN can leave a process hit. Build a matrix for each data class: cache layers, key schema, fresh lifetime, maximum stale period, writer, invalidation signal, and behavior when a layer fails.

Publish invalidation only after the database transaction commits. An outbox or equivalent mechanism avoids the gap where data commits but its event disappears. Consumers must be idempotent and reject an older out-of-order event using the entity version. A versioned namespace can be safer than enumerating deletes: switching catalog:v42:... to v43 makes new reads bypass the old set immediately, while old keys expire naturally. This does not replace capacity management, but it separates correctness from the success of a bulk deletion.

The write order also matters. A common pattern deletes the cache after a successful database commit so the next read rebuilds it. Deleting before the write allows a concurrent reader to refill the old database value. Strongly consistent reads should bypass the cache or use version conditions; an eventually consistent cache cannot honestly provide transaction semantics.

Make every outcome explainable

Record hit, miss, stale, revalidated, bypass, and error, along with layer, key-schema version, object version, age, and rebuild duration—never the sensitive full key. RFC 9211 defines Cache-Status so HTTP caches can explain how they handled a response. Multiple layers append members in traversal order, making it easier to identify which one returned old content.

Do not build a dashboard around hit rate alone. Track origin calls, coalesced waiters, rebuild failures, stale-response age, evictions, key cardinality, and origin latency. A high hit rate can hide persistently wrong data; a low one might simply reflect traffic that cannot safely be reused.

Tests should prove that tenants and authorization scopes never cross, Vary dimensions produce different representations, TTL boundaries and 304 updates work, origin errors respect the stale window, concurrent hot requests perform one rebuild, invalidation events tolerate duplication and reordering, and one unavailable cache layer degrades safely. Inject a clock instead of sleeping through real TTLs.

Good caching is not about storing as much as possible. It places a bound on stale-data risk, coalesces origin pressure, converges across layers after change, and explains every reuse. Once those conditions are explicit, a cache becomes a reliability tool instead of a second database hidden behind successful responses.