Surviving failure
A queue hands work to a worker. A log lets you replay it.
Redelivery recovers one unfinished task; replay rebuilds a consumer from retained history.
An independent note, with worked examples. Watch the source lessons.
Retry and replay answer different questions
A notification worker receives job notify-8042, sends an SMS, and crashes before acknowledging the job. The immediate question is whether another worker should try that task. Six hours later, an operator fixes a bug in the notification-status projection. The question is now whether the system can run every relevant event through the corrected code again.
The first problem needs redelivery. The second needs retained history and a movable consumer position. Both are sometimes described as “put it on a queue,” but they require different storage and consumption contracts.
Arpit Bhayani’s Razorpay notification walkthrough separates notification work in SQS from status updates flowing through Kinesis and a retry scheduler. The example is useful because the work handoff and the event history serve different recovery paths.
Name what happens after a consumer reads
Hello Interview’s message-queue lesson emphasizes acknowledgements, visibility timeouts, redelivery, poison messages, and idempotent consumers. Those details define a work queue more clearly than its product name.
| Question | Work queue | Retained partitioned log |
|---|---|---|
| Who handles an item? | Leased work; duplicate delivery must be tolerated. | One consumer per partition in each consumer group. |
| What marks progress? | Acknowledgement or deletion. | A committed offset per group. |
| After success? | The item may disappear. | The record remains until retention removes it. |
| Can another application read it? | Only if separately copied or routed. | A separate group reads the same record independently. |
| Can old work be rerun? | Not assumed after acknowledgement. | Yes, while the record is retained and offsets can move. |
These are archetypes, not a vendor checklist. Some brokers support both patterns. Record the needed behavior explicitly instead of inferring it from the word queue.
The worker crash creates a duplicate, not certainty
Return to notify-8042. Worker A receives a lease, asks the SMS provider to send, and the provider accepts the request. A crashes before acknowledging the queue or recording success. When the visibility timeout expires, worker B receives the same job.
stable operation id: notify-8042
worker A: provider accepts → crash
queue: lease expires → redeliver
worker B: check/send using notify-8042 → acknowledgeThe broker cannot know whether the external side effect happened. A stable operation ID, a local attempt ledger, and a provider idempotency facility where available let B recover the unknown outcome. Without them, at-least-once delivery can become at-least-twice sending.
Acknowledging before the send avoids the duplicate but can lose the notification after a crash. Acknowledging after the send avoids that loss window but requires duplicate handling. “Exactly once” is not obtained by changing the order of two non-atomic operations.
The same rule applies to the status stream: update the projection, then commit the consumed offset. If a crash repeats the update, the handler must make the repeated event harmless.
Replay starts from an older offset
Hello Interview’s Kafka deep dive describes append-only topic partitions, consumer groups, offsets, replication, keys, and retention. That model lets a corrected projection use a new consumer group or reset an existing group to an earlier offset and process retained records again.
Replay the projection separately from the delivery consumer: rebuilding a status table should not resend old notifications. Replay is broader than retry. A retry asks the operational system to attempt one failed item again, often soon and with a limit. A replay intentionally revisits many already successful records to rebuild state, add a new consumer, or repair faulty derived data.
Replay works only for data still inside the retention horizon, and only if the records contain enough immutable facts to recompute the result. A log containing “counter is now 12” may not repair as cleanly as one containing the stable event that changed it. Schema versions, deterministic handlers, event-time rules, deletion semantics, and access to referenced data all affect whether history remains usable.
A dead-letter queue is not automatically that history. It contains selected failures, not necessarily every event needed to rebuild the projection.
Design the recovery path before choosing the broker
Use a work queue when the unit is a task that one worker should lease, finish, and acknowledge. Define the visibility timeout, retry budget, dead-letter handling, backpressure, and idempotency key. Use a retained log when several consumers need the same ordered history or a projection must be rebuildable. Define the partition key, ordering scope, retention horizon, offset policy, and replay procedure.
Many systems need both. A durable outbox can publish a notification-requested event. One consumer creates delivery work; another builds an audit view. Delivery attempts can append result events for status and analysis. The task queue absorbs worker churn, while the log preserves the facts needed by independent consumers.
For a review exercise, crash the worker before the provider call, after provider acceptance, and after acknowledgement. Then delete the status projection and rebuild it. Record the durable state used to recover at each pause, and verify that replay rebuilds the expected projection.