The cache was invalidated. Then an old value came back.

Revocation makes the race visible

A guest cancels reservation 500. The database removes that guest’s right to read listing 1, and the authorization service invalidates its cached decision. A moment later, the cache again says the guest may read the listing.

This can happen without a lost write or a failed invalidation. Arpit Bhayani’s Himeji walkthrough describes a cache in front of authorization data, with database changes flowing through CDC and Kafka to an invalidation worker. That is a useful way to keep reads fast, but the asynchronous path creates a window that a cache-aside reader can cross.

Hello Interview’s Redis deep dive presents cache-aside reads and expiration as common Redis patterns. Put the two ideas together and a less obvious failure appears: deleting a stale entry does not stop an older read from filling it again.

The invalidation arrives between read and fill

Use one concrete key, auth:listing:1:user:456. Its database value starts at allow, version 41.

The invalidator did exactly what its event requested. It could not distinguish an empty cache from a cache fill still in flight. This is the stale-fill race.

TTL starts when the stale value is inserted

Suppose step 4 writes the result with a 60-second TTL. That limits the entry’s lifetime after the fill; it does not prove that the value is at most 60 seconds older than the database.

The database read occurred at step 1. The reader might then wait behind a network pause, a saturated connection pool, a stop-the-world pause, or a retry. If it resumes much later, Redis still starts a fresh 60-second countdown when it inserts v41. The value can therefore be old before its TTL begins.

Even a short fill normally permits some stale service after a concurrent write. That can be acceptable for a product description and unacceptable for access revocation. State the business contract before choosing the mechanism: eventual convergence, a numerical staleness budget, read-your-writes, or immediate revocation are different promises.

Make an old fill prove that it is still current

One repair adds a monotonic generation beside each logical key. A reader records generation 7 before loading the database. The invalidation for v42 advances the generation to 8 and deletes the cached value. The reader may fill only with a conditional operation equivalent to:

if current_generation(key) == 7:
    set_cached_value(key, allow_v41, ttl=60s)
else:
    discard_and_retry()

In the worked sequence, the old reader loses the comparison and cannot resurrect v41. The comparison and fill must be atomic with respect to advancing the generation. An in-process check followed by a separate SET merely creates another race. Keep the generation record for longer than any possible in-flight fill; eviction or reuse of an older generation can admit an obsolete reader again.

This closes the race only after the generation advance arrives. If the invalidation stream itself is delayed, a stale value can still be served during that delay. A safety-critical revocation may need an authoritative version check, a synchronous write-through path, or a cache bypass until the new version is visible.

A weaker but useful guard rejects a fill when too much time has elapsed since the database read. If a fill must finish within 500 milliseconds, a 60-second TTL yields a stated bound of roughly the fill window plus the TTL, provided the source read was current at its observation time, elapsed time is measured correctly, and cache hits do not extend expiry. It bounds the exposure rather than proving freshness.

Reproduce the interleaving

A cache design review should name the source of truth, the invalidation trigger, the maximum propagation delay, and what a reader does when the cache and source disagree. It should also say whether negative decisions are cached and whether fail-open behavior is ever allowed.

Then pause a cache-miss reader immediately after its database read. Commit a newer value, deliver the invalidation, resume the reader, and inspect the next read. The test fails the naive delete-and-fill design and passes only when the older fill is rejected or safely bounded.

TTL remains useful for cleanup and eventual convergence. It is not, by itself, evidence that a cached authorization decision is fresh enough.