Queues and Messaging, In Depth

Async messaging means one part of a system hands off a piece of work or a fact by writing a message — a small self-contained record of "something to do" or "something that happened" — into a middle store, instead of calling another part directly and waiting for it to finish. The part that writes is the producer; the part that reads and acts is the consumer; the middle store that holds messages between them is the broker (a server whose whole job is to accept, store, and hand out messages). The producer does not need the consumer to be awake, fast, or even alive at the moment it sends.

That one hop of indirection buys four things at once:

• Decoupling. The producer depends on the broker, not on the consumer's address, health, or speed. Either side can be deployed, restarted, or scaled without the other noticing.
• Buffering bursts. When traffic spikes, the broker absorbs the flood. Messages pile up safely and drain at whatever rate consumers can manage, instead of overwhelming a slow downstream service.
• Retries. If processing fails, the message is still in the broker, so it can be delivered again. A dropped direct call is gone; a queued message is not.
• Load smoothing. Slow work (sending email, generating a thumbnail, charging a card) moves off the request path. The user's request returns immediately after the message is written; the heavy lifting happens later.

The cost is honesty about a new reality: the work is now asynchronous, it can be delivered more than once, it can arrive out of order, and it can fail after the user already got a success. The rest of this day is the toolkit for living with that reality safely.

There are two fundamentally different shapes of messaging system, and picking the wrong one causes most messaging pain. The first is the point-to-point queue: a message is delivered to one consumer, processed once, and then removed. The second is the partitioned append-only log (also called a stream): messages are appended to an ordered file that is never edited, and many independent readers each keep their own place in it.

A queue is like a shared to-do inbox. Any free worker grabs the next item, does it, and throws the slip away — once it is done, it is gone.

A log is like an ever-growing ledger that is never erased. Each reader remembers the line number it last read and comes back later to continue. Because nothing is deleted on read, a brand-new reader can start from line one and replay the entire history.

The vocabulary that makes the log model work:

• Partition — a log is split into several independent ordered sub-logs called partitions, so it can be spread across machines and read in parallel. Order is guaranteed within a partition, not across partitions.
• Offset — the position number of a message inside a partition. Message 0, 1, 2, and so on. A reader's progress is simply "the next offset I have not yet read."
• Consumer group — a named set of consumers that cooperate to read a log once between them. The broker divides the partitions among the group's members, and it tracks one set of offsets per group. Two different groups reading the same log are independent: each gets every message, at its own pace.

Queue-style brokers (Amazon SQS, RabbitMQ in its classic mode) shine when work is a set of independent tasks and you never need to re-read the past. Log-style brokers (Apache Kafka, Amazon Kinesis, Redpanda) shine when several different systems each need the full stream, when you want to replay history, and when per-key ordering matters. Here is the direct comparison:

Rule of thumb: if each message is a job to be done once and forgotten, reach for a queue; if the same events must feed several independent systems or be replayable, reach for a log.

"Delivery semantics" is the promise a messaging system makes about how many times a message reaches its consumer. There are three phrases you will hear, and understanding them prevents a whole category of data-loss and double-charge bugs. Two are real guarantees the infrastructure can give you; the third is a goal you assemble yourself.

• At-most-once. The broker sends the message and immediately forgets it. If the consumer crashes before finishing, the message is simply lost — it is never redelivered. You get zero or one processings. This is the cheapest and fastest, and acceptable only when losing a message now and then does no harm (high-volume metrics, best-effort telemetry).
• At-least-once. The broker keeps the message until the consumer explicitly confirms success (the ack, short for acknowledgement — a signal from consumer back to broker meaning "I have safely finished this one, you may delete it"). If no ack arrives within a time window, the broker assumes failure and delivers the message again. You get one or more processings — never zero, but possibly duplicates. This is the workhorse default for anything that matters.
• Exactly-once. Every message takes effect exactly one time — no loss, no duplicate. This is what everyone wants, and it is where the trap lies.

Here is the key truth: true end-to-end exactly-once delivery is an illusion. The network can drop the ack after the work is done, so the broker cannot tell "succeeded but ack lost" apart from "failed," and it must redeliver to be safe. No amount of broker cleverness removes that ambiguity across a real network. What you actually build is exactly-once effect, assembled from three parts:

At-least-once delivery guarantees you never lose a message; the idempotent consumer and a dedupe key guarantee that processing the same message twice changes the world only once. The combination gives the outcome of exactly-once without ever needing the broker to promise it. When a vendor advertises "exactly-once," it is doing exactly this internally within a bounded scope — never magic.

Rule of thumb: choose at-least-once delivery and make the consumer idempotent; treat "exactly-once" as a property you engineer at the consumer, not a checkbox on the broker.

Ordering is the promise that messages are processed in the sequence they were sent. It sounds like something you always want, but strict ordering and high throughput pull against each other, so you buy only as much ordering as the data actually needs. There are two granularities worth naming.

Global ordering means every message across the whole system is processed in one strict sequence. That requires a single point that serializes everything — effectively one consumer working alone — because the moment two consumers run in parallel, their relative timing is unpredictable. Global ordering therefore caps throughput at what one worker can do.

Per-partition or per-key ordering is the practical middle ground: messages that share a key (say, all events for account-123) are guaranteed to be handled in order relative to each other, while unrelated keys flow in parallel. The producer routes a message to a partition by hashing its key, so the same key always lands in the same partition, and a partition is read in strict offset order.

The tradeoff is precise: ordering constrains parallelism. If everything must be globally ordered, you can run only one consumer and throughput is capped. If only same-key events must be ordered, you get ordering and parallelism, limited by how many distinct keys and partitions you have. Choosing a good key — one with many distinct values and no single dominant "hot" value — is what keeps a log both correct and fast.

Rule of thumb: never ask for global ordering unless the domain truly demands it; pick a partition key so that only events which must be ordered together share a key.

Throughput on a log scales by adding partitions and adding consumers to a group — but there is one hard rule that governs the whole picture: within a consumer group, each partition is read by at most one consumer at a time. The broker assigns partitions to group members so that no two members ever process the same partition simultaneously (that is what preserves per-partition order).

Two consequences follow directly from that rule:

• The partition count is your parallelism ceiling. A group can usefully run at most as many consumers as there are partitions. Four partitions means at most four working consumers in a group; a fifth consumer sits idle with nothing assigned to it. If you expect to scale to eight workers later, create eight partitions now, because raising the partition count afterward is disruptive and can scramble key-to-partition mapping.
• Rebalancing happens automatically. When a consumer joins or dies, the broker reshuffles partition assignments across the surviving members. Briefly, processing pauses while the group settles on who owns what, then resumes from each partition's last committed offset.

A plain point-to-point queue scales more simply — you just add workers to the shared pool and the broker hands each free worker the next message — but you give up the per-key ordering guarantee that partitions provide. Rule of thumb: size the partition count for your peak future parallelism, since one partition can feed only one consumer per group.

Backpressure is what a system does when work arrives faster than it can be processed. Without a deliberate answer, a fast producer and a slow consumer end up filling memory until something crashes. Flow control is the set of mechanisms that keep the fast side from overwhelming the slow side — the messaging equivalent of a valve.

The core tool is a bounded queue: a buffer with a fixed maximum size. When it fills, the system must make a choice, and there are only a few honest options:

• Pause and resume. The consumer tells the broker (or the producer) to stop sending until it has caught up, then signals resume. This is the cleanest backpressure: the slow side sets the pace, and no work is lost — it just waits at the source.
• Reject / shed load. When the buffer is full, refuse new work outright and return a fast "try again later" (for example an HTTP 429). Rejecting is often safer than buffering: an unbounded buffer just moves the failure to a worse place (out-of-memory) and hides the overload until it is catastrophic. A visible, immediate rejection lets clients back off.
• Buffer more. Grow the buffer (bigger queue, spill to disk). This absorbs a short spike but only delays the reckoning if the overload is sustained — you cannot buffer your way out of a consumer that is permanently too slow.

The load-bearing insight: an unbounded buffer is not flow control, it is a delayed crash. Bound your queues, decide in advance whether a full queue blocks or rejects, and make the limit visible so overload is a handled condition rather than a surprise. Rule of thumb: prefer bounded queues that pause the producer or reject excess over unbounded buffering that fails as an out-of-memory crash under sustained load.

When processing a message fails, the message is still in the broker, so the natural response is to try again. But naive retrying — immediately, forever, all at once — turns a small hiccup into an outage. Two techniques make retries safe: exponential backoff and jitter, plus a hard limit.

Exponential backoff means each successive retry waits longer than the last, typically doubling: 1s, 2s, 4s, 8s, 16s. A struggling downstream service gets breathing room to recover instead of being hammered by a wall of instant retries.

Jitter adds a small random amount to each wait. Without it, many consumers that failed at the same instant would all retry at the same instant — a synchronized "retry storm" that repeatedly slams the recovering service in waves (the thundering-herd problem). Randomizing the delay spreads the retries out so the load is smooth.

attempt 1: fail, then wait ~1s  (1s base plus random 0..1s)
attempt 2: fail, then wait ~2s  (2s base plus random 0..2s)
attempt 3: fail, then wait ~4s  (4s base plus random 0..4s)
attempt 4: fail, then wait ~8s  (8s base plus random 0..8s)
attempt 5: fail, then give up, send to dead-letter queue

The retry limit is the third essential: a maximum number of attempts (or a maximum total age). Retrying forever means a message that can never succeed — a bad payload, a permanently missing record — blocks a consumer indefinitely and burns resources. After the limit, the message must be set aside rather than retried again, which is exactly what the dead-letter queue in the next section is for.

Rule of thumb: retry with exponential backoff plus random jitter, and always cap the attempts — unlimited instant retries amplify failures instead of recovering from them.

A poison message is a message that fails every time it is processed, no matter how often it is retried — a malformed payload, a reference to a record that was deleted, a bug that only that message triggers. Left alone, a poison message is redelivered forever, blocking the queue behind it and wasting compute on a hopeless task. The standard remedy is the dead-letter queue.

A dead-letter queue (DLQ) is a separate queue where messages are moved after they exhaust their retry limit. It is a quarantine: the message is taken out of the main flow so healthy messages keep moving, and it is preserved (not dropped) so a human or a repair job can inspect it, understand why it failed, and decide what to do.

Why the DLQ matters:

• It unblocks the main flow. One poison message can no longer stall everything behind it — after N failures it steps aside into the DLQ.
• It preserves evidence. The failed message is not lost; you can read it, see the error, and diagnose the root cause.
• It enables recovery. Once the bug is fixed, DLQ messages can be replayed back into the main queue and processed for real.

A DLQ that nobody watches is a silent data-loss bucket, so alerting on DLQ depth is part of the pattern — a rising DLQ count is one of the earliest signals that something downstream is broken. Rule of thumb: give every important queue a dead-letter queue with a retry limit, and alert on its depth so poison messages surface instead of silently piling up or blocking healthy work.

This section explains the exact mechanics that make at-least-once delivery work in a point-to-point queue. The moving parts are the visibility timeout, the ack, and redelivery — three cooperating ideas that let a broker guarantee a message is processed even if consumers crash.

When a consumer receives a message, the broker does not delete it. Instead it hides the message from other consumers for a window called the visibility timeout — a countdown during which the message is invisible to everyone else, giving this consumer exclusive time to finish. Two things can happen next:

• The consumer succeeds and sends an ack. The ack (acknowledgement) tells the broker "safely done," and only then does the broker delete the message. No ack, no deletion — that is what guarantees nothing is lost on a crash.
• The consumer crashes, hangs, or is too slow. The visibility timeout expires with no ack, so the broker concludes the consumer failed and makes the message visible again for redelivery to another consumer.

This design is exactly why duplicates are unavoidable. Picture a consumer that finishes the real work — charges the card — and then crashes in the microsecond before its ack reaches the broker. From the broker's point of view no ack arrived, so it redelivers, and a second consumer charges the card again. The work succeeded once but is delivered twice; the broker had no way to know. This is the concrete reason every consumer must be idempotent.

The visibility timeout must be tuned: too short and a slow-but-healthy consumer gets its message yanked away and processed twice needlessly; too long and a genuine crash leaves the message stuck, invisible and unprocessed, until the long timeout finally elapses. For work whose duration varies, many brokers let a consumer extend the timeout (a "heartbeat") while it is still making progress. Rule of thumb: set the visibility timeout a bit longer than your realistic worst-case processing time, and design for the fact that a crash between doing the work and sending the ack will cause a redelivery.

An operation is idempotent when doing it twice leaves the world in the same state as doing it once. Because at-least-once delivery, visibility-timeout redelivery, and retries all guarantee that some messages arrive more than once, every consumer must tolerate duplicates — idempotency is not an optional nicety, it is the price of admission for reliable messaging.

The practical mechanism is a dedupe key: a stable identifier carried by the message that names the specific effect it represents (an order id, a payment id, an event id — not a random per-delivery id, which would differ on each redelivery). Before acting, the consumer checks whether it has already handled that key; if so, it skips the work and just acks.

Two implementation styles cover most cases:

• Check-then-act with a processed-keys table. Keep a table of keys you have already handled. On each message, look up the key; if present, skip; if absent, do the work and insert the key. To make this race-safe, do the "insert the key" and "do the work" inside one database transaction, or rely on a unique constraint on the key so a duplicate insert fails loudly instead of double-acting.
• Naturally idempotent writes. Sometimes you can shape the operation so repetition is harmless by construction — SET status = 'shipped' (running it twice still yields "shipped"), or an upsert keyed by the message's id. When the write itself is idempotent, no separate bookkeeping is needed.

The subtle failure to avoid: acking before the work is durably committed. If you ack first and then crash, the message is deleted but the effect never happened — you have silently lost it. Always finish (and persist) the work, then ack. Rule of thumb: give every message a stable dedupe key and make each consumer check-then-act (or write idempotently), because the delivery layer will hand you duplicates.

A very common need is: change the database and tell other services about the change, as one indivisible act. The obvious approach — write the row, then publish an event to the broker — is a dual write, and it is quietly broken. The two systems (database and broker) cannot commit together, so any crash in the gap between them corrupts your data.

Look at what the crash does: the order exists in the database, but no event was ever published, so the shipping service, the email service, and the analytics pipeline never find out. The row and the event have drifted apart, and nothing will ever reconcile them. (Swapping the order — publish first, then write — is no better: now you can publish an event for an order that the database crash prevented from ever existing.) A dual write across two systems that cannot share a transaction will lose messages on partial failure.

The outbox pattern fixes this by removing the second system from the critical moment. Inside the same database transaction that writes the business row, you also insert a row into an outbox table — a table that holds events waiting to be published. Because both writes are in one transaction, they commit together or not at all; there is no gap to crash in. A separate background worker, the relay (also called a publisher or dispatcher), then reads unpublished outbox rows, publishes them to the broker, and marks them published — retrying until each one lands.

Why this is correct where the dual write was not:

• Atomicity is real. The business row and its event share one transaction, so you can never have one without the other. A crash either rolls both back or commits both.
• The relay is at-least-once by design. If the relay publishes but crashes before marking the row published, it simply republishes on restart — which is fine, because downstream consumers are idempotent and dedupe on the event id. The outbox produces duplicates, never losses.
• The broker's health is decoupled from the request. If the broker is down when the order is placed, the user's request still succeeds; the event waits safely in the outbox and is delivered whenever the broker returns.

The outbox pattern is the standard fix for this; here the important detail is why the naive alternative fails (two systems, no shared commit, a lossy gap) and why the relay's duplicate publishes are harmless (idempotent, deduped consumers). Rule of thumb: never dual-write to a database and a broker; write the event to an outbox table in the same transaction as the business row, and let a relay publish it at-least-once.

Everything so far handed each message to one worker or one consumer group. Fan-out is the opposite need: one event should reach many independent subscribers, each doing its own thing. The pattern that provides this is publish/subscribe (pub/sub).

In pub/sub, a producer (the publisher) sends a message to a named channel called a topic — a labeled stream that subscribers register interest in — rather than to a specific consumer. The broker then delivers a copy of every message on that topic to each subscriber. The publisher does not know or care how many subscribers exist; adding a new one requires no change to the publisher.

The defining property is one message, many independent copies. Contrast it with a plain queue, where one message goes to exactly one worker:

The log model unifies these: a Kafka-style topic with partitions gives fan-out (each consumer group is an independent subscriber that receives everything) while still giving work distribution within a group (partitions split across the group's members). So one log can feed the email service, the shipping service, and analytics as separate groups, and each of those groups can internally scale across several workers.

A key benefit of fan-out is loose coupling over time: when a new capability appears — say a loyalty-points service — it simply subscribes to the existing order-created topic and starts receiving events, with zero changes to the order service. Rule of thumb: when one event must trigger several unrelated reactions, publish it to a topic and let each interested service subscribe, instead of having the producer call each consumer directly.

This table collapses the day into problem-to-tool pairs. Each defense is aimed at a specific failure that async messaging introduces.

• Async messaging inserts a broker between producer and consumer to decouple them, buffer bursts, enable retries, and smooth load — at the price of asynchrony, duplicates, and reordering.
• A point-to-point queue delivers each message to one consumer and deletes it; a partitioned log keeps messages so many consumer groups can read and replay them via offsets.
• Delivery is at-most-once or at-least-once; "exactly-once" is an end-to-end illusion built from at-least-once delivery plus idempotent, deduped consumers.
• Ordering costs parallelism: global order needs one consumer, so partition by key and order only what must be ordered together.
• A partition feeds at most one consumer per group, so the partition count is your parallelism ceiling — size it for peak future load.
• Bound your queues and choose pause-or-reject; an unbounded buffer is a delayed out-of-memory crash, not flow control.
• Retry with exponential backoff and jitter under a hard limit; send exhausted poison messages to a dead-letter queue and alert on its depth.
• The visibility timeout hides a message until its ack; a crash before the ack causes redelivery, which is exactly why every consumer must be idempotent with a stable dedupe key.
• Never dual-write to a database and a broker; use the outbox pattern so the row and its event commit together and a relay publishes at-least-once.
• For one-to-many delivery, publish to a topic and let each service subscribe, keeping producers unaware of their consumers.