Publish/Subscribe (Pub/Sub) in System Design: Topics, Fan-Out & Delivery Guarantees (Visualized)

Publish/subscribe (pub/sub) is a messaging pattern in which senders, called publishers, emit messages to a named channel called a topic, and receivers, called subscribers, register interest in that topic to receive copies of those messages. The defining property is decoupling: a publisher does not know who (if anyone) is subscribed, and a subscriber does not know who produced the message.

This indirection is what makes pub/sub the backbone of event-driven architectures. A single "order placed" event can fan out to a billing service, an email service, an analytics pipeline, and a fraud detector at once — and you can add a fifth consumer later without touching the publisher. The broker in the middle handles routing, buffering, and delivery.

Three roles make up the pattern. Publishers produce messages and address them to a topic, never to a specific recipient. The topic is a logical channel managed by a broker (Kafka, Google Pub/Sub, SNS, Redis). Subscribers declare interest in one or more topics; the broker delivers a copy of every matching message to each subscriber. Because all communication flows through topics, publishers and subscribers can be deployed, scaled, and restarted independently. This strict separation of concerns means you can add a new data consumer — say, a fraud-detection service — to an existing event stream with zero changes to any existing publisher or subscriber. The broker is the only shared contract, and that contract is simply a topic name and a message schema.

The signature behavior of pub/sub is fan-out: when a publisher sends one message to a topic, every subscriber to that topic receives its own copy, simultaneously and independently. There is no contention between subscribers — each gets the full stream. This is exactly what you want when many distinct services must react to the same event.

It is easy to confuse pub/sub with a plain message queue, but they distribute work differently. In a point-to-point queue, each message is delivered to exactly one consumer; multiple consumers compete and the broker load-balances among them. In pub/sub, each message is delivered to every subscriber. Queues spread work; topics broadcast events. A useful mental model: think of a message queue as a shared task list where only one worker picks up each item, and a pub/sub topic as a newspaper — every subscriber gets their own copy of every edition, regardless of how many others also read it.

Brokers deliver messages to subscribers in one of two modes. With push, the broker actively sends each message to the subscriber (for example an HTTP POST to a webhook, as Amazon SNS does). With pull, the subscriber polls the broker and fetches messages at its own pace (Kafka consumers and Google Pub/Sub pull subscriptions). Push is low-latency but can overwhelm a slow consumer; pull gives the consumer back-pressure control and lets it batch. In high-throughput systems, pull is almost always preferred: a subscriber can read in batches of hundreds or thousands of messages per request, dramatically reducing round-trip overhead, and it can pause consumption entirely during a maintenance window without the broker queueing up retries. Google Cloud Pub/Sub supports both modes simultaneously — the same subscription can be configured as a push endpoint or polled via gRPC streaming pull.

Subscribers often want only some messages on a topic. Filtering lets the broker evaluate predicates against message attributes and deliver only matches. SNS supports message-filtering policies on attributes; Google Cloud Pub/Sub supports attribute and CEL-based body filters; many MQTT brokers use hierarchical topic wildcards like sensors/+/temperature. Filtering moves the discard decision to the broker, saving the subscriber from receiving and dropping irrelevant traffic. This is especially valuable in high-fan-out scenarios: if a topic receives 100,000 events per minute but a given subscriber only cares about events where region == "us-east", the broker can discard the other 95 % before they ever cross the network to that subscriber. Done at scale, server-side filtering cuts both egress cost and subscriber CPU load significantly.

Brokers offer different delivery guarantees: at-most-once (fire and forget, messages may be lost — Redis classic pub/sub falls here), at-least-once (retried until acknowledged, so duplicates are possible — Kafka and Google Pub/Sub default), and exactly-once (no loss, no duplicates — the hardest and most expensive, available in Kafka with idempotent producers and transactional consumers, and in Google Cloud Pub/Sub's exactly-once subscriptions). Because at-least-once is the common default, consumers should be idempotent so reprocessing a duplicate is harmless. A practical technique is to assign a unique message ID at publish time and track processed IDs in a fast store like Redis; if the ID is already in the set, skip processing and ack immediately. This keeps your consumers correct under retries without needing expensive distributed transactions.

Ordering is a separate concern. Global ordering across an entire topic is rarely guaranteed at scale. Kafka preserves order only within a partition; messages sharing a partition key arrive in order, while different keys may interleave. Google Pub/Sub offers ordering keys with the same trade-off. Design your keys so that messages that must be ordered (e.g. all events for one user) share a partition.

The line blurs in practice. Kafka combines both: a topic broadcasts to multiple consumer groups (pub/sub fan-out), while within a single group partitions are split among members (queue-style work distribution). RabbitMQ models pub/sub with a fanout exchange in front of per-consumer queues.

A simple rule of thumb: choose a queue when you have a single logical job that many workers should share, and choose pub/sub when one event must trigger several unrelated reactions. When you need both — durable history and broadcast — reach for a log-based broker like Kafka, where each consumer group keeps its own offset and reads the full stream independently.

Apache Kafka is a durable, partitioned append-only log: subscribers pull, messages are retained for a configurable window (days to weeks), and each consumer group tracks its own read offset independently of every other group. This makes Kafka the broker of choice when new consumers must replay historical data or when multiple teams each need the full event stream without any coordination overhead. Throughput can reach millions of events per second across a well-tuned cluster. Amazon SNS is a serverless, managed push pub/sub that fans out to SQS queues, Lambda functions, HTTP/HTTPS endpoints, email, and SMS — with JSON attribute filtering so only relevant subscribers receive each message. It pairs naturally with SQS in the SNS+SQS fan-out pattern, where one SNS topic feeds N independent queues that each buffer work for a different downstream system. Google Cloud Pub/Sub is a globally distributed managed broker with both push and pull delivery, at-least-once and exactly-once subscription modes, dead-letter topics, and ordering keys. Redis Pub/Sub is in-memory and fire-and-forget (at-most-once) — sub-millisecond latency but no persistence; if no subscriber is connected when a message is published, that message is lost. Redis Streams adds consumer group semantics and durability to Redis, making it far more suitable for production event pipelines that need replay or guaranteed delivery.

A message queue delivers each message to exactly one consumer, so adding consumers spreads the work. Pub/sub delivers a copy of each message to every subscriber, so adding a subscriber gives you another independent stream. Use a queue to parallelize processing of one job stream; use pub/sub to broadcast an event to many distinct services.

Usually not globally. Most brokers guarantee ordering only within a partition or ordering key — messages sharing that key arrive in order, but messages across keys may interleave. If end-to-end order matters, route related messages through the same key, and remember that at-least-once delivery still means consumers should be idempotent.

Both. Kafka topics fan out to multiple consumer groups like pub/sub, while members within one consumer group split the topic's partitions like a work queue. Its durable, replayable log also lets late or new subscribers read past messages — something classic in-memory pub/sub such as Redis cannot do.