Message Brokers vs Event Buses

The claim under test

An estate publishes domain events onto Redis channels. Services subscribe by pattern. No service imports another service's classes. The architecture document says event-driven, and by one reasonable definition it is.

11 services · 0 brokers

channels     mission.*.areas          area create / update / delete
             mission.*.entities       entity position updates
             mission.*.geofencing     geofence breach notifications

publish      redisTemplate.convertAndSend(channel, payload)
subscribe    implements MessageListener   (Spring Data Redis)

guarantees   persistence  ✗      replay  ✗      consumer groups  ✗
             broker ack   ✗      DLQ     ✗      backpressure     ✗

The decoupling is real. The publisher genuinely does not know who is listening, which is the property most teams are actually buying when they reach for a bus. What it does not buy, and what the word "event-driven" quietly implies, is that the event will arrive. Fowler opened his 2017 write-up of an event-driven summit with the reason this argument keeps recurring:

The pattern above is his event notification: "a system sends event messages to notify other systems of a change in its domain", and, the part that matters here, "the source system doesn't really care much about the response". That is a coherent, useful pattern. It is only a problem when the receiver treats the notification as the only copy of the fact.

What Redis pub/sub actually guarantees

Redis is unusually honest about this, and the sentence to internalise is theirs, not a critic's. It is in the delivery-semantics section of the pub/sub documentation:

Read that mechanically rather than as a caveat. PUBLISH is a routing operation over sockets that are open at that instant. There is no store to write to, so there is nothing for a late subscriber to read; the documentation notes that "Pub/Sub has no relation to the key space": the channel is not a key, it is a fan-out table. Redis returns the number of clients it wrote to, which is a fan-out count and not a delivery receipt: it does not tell you whether any of those clients finished processing, and it cannot tell you about the subscriber that reconnects a second later.

The one guarantee that is offered is ordering: "Subscribers receive the messages in the order that the messages are published". That is worth naming, because ordering without delivery is a weaker property than it sounds. You get the surviving messages in order, with unobservable gaps where the rest used to be.

That gives you the decision, and it is not "Redis or Kafka". It is: is this channel carrying a notification, or is it carrying the only copy of a fact?

Delivery semantics, precisely

Three terms get used loosely and one of them is routinely oversold. Kafka's design documentation states them in a single line each, and the wording is worth taking literally. Note that the second says never lost, not delivered once.

1. At-most-once

   "Messages may be lost but are never redelivered." The sender does not wait, the receiver does not confirm, and nothing retries. Redis pub/sub sits here permanently, not by configuration, but because there is no store that could serve a retry.

   Acceptable when the next message supersedes this one. Unacceptable the moment a receiver accumulates state from the stream.

2. At-least-once

   "Messages are never lost but may be redelivered." The broker holds the message until a consumer acknowledges it; an unacknowledged message is redelivered after a timeout, a nack, or a consumer crash. This is what a durable queue or a log with committed offsets gives you, and it is the strongest guarantee a broker can offer on its own.

   The cost is duplicates, and they are not rare. Every consumer restart mid-batch produces them. That cost is transferable: make the consumer idempotent and duplicates stop mattering.

3. Exactly-once, and why it is not a delivery guarantee

   Kafka's own definition is "Each message is processed once and only once", and note processed, not delivered. Delivery cannot be exactly-once across a network partition: the sender either retries an unacknowledged send (risking a duplicate) or does not (risking a loss), and it cannot distinguish "lost request" from "lost response".

   What real systems ship is at-least-once delivery plus idempotent consumption, usually called effectively-once. Kafka's idempotent producer removes duplicates on the produce path, and transactions give atomic read-process-write when the source and the sink are both Kafka. The instant your consumer writes to a relational database or calls a third-party API, you are back to at-least-once and the deduplication is yours to build.

Kafka also names its own default plainly: it "guarantees at-least-once delivery by default, and allows the user to implement at-most-once delivery by disabling retries on the producer and committing offsets in the consumer prior to processing a batch of messages". Committing before you process is the one-line difference between the two guarantees, and it is a configuration mistake people make by accident.

Four buses, eight properties

"Use Kafka" is not the answer to most of these problems, and it is worth seeing why: Redis Streams closes almost the whole gap without adding a second piece of infrastructure. Read the amber cells carefully, because they are where the marketing and the mechanism disagree.

The failure mode, drawn

The whole lesson is one comparison: what the system does with a message whose consumer is not there. On the left there is nowhere to put it. On the right there is.

What this looked like in production

Doing it properly

Three tabs, in the order the problem is usually met. The first is the fire-and-forget publisher: correct code, with one property worth reading carefully. The second moves the same event onto a durable queue with manual acknowledgement and a broker-routed dead letter. The third is the one that actually matters: at-least-once is only an improvement if applying an event twice is indistinguishable from applying it once.

@Slf4j
@Service
@RequiredArgsConstructor
public class AreaEventPublisher {

    /** The only collaborator. Nothing in this class knows a subscriber exists. */
    private final RedisTemplate<String, Object> redisTemplate;

    public void publishAreaChanged(String missionId, AreaChanged event) {
        String channel = "mission." + missionId + ".areas";

        // convertAndSend returns the number of client sockets Redis wrote to.
        // That is a fan-out count, not a delivery receipt: it says nothing about
        // whether any consumer finished handling the event, and the subscriber
        // that reconnects one second from now is not counted and never will be.
        Long receivers = redisTemplate.convertAndSend(channel, event);

        log.debug("published {} to {}, {} socket(s)", event.eventId(), channel, receivers);
    }
}

@Configuration
public class AreaMessagingConfig {

    static final String QUEUE = "mission.areas.projection";
    static final String DLX   = "mission.areas.dlx";
    static final String DLQ   = "mission.areas.dlq";

    /**
     * durable(...) means the queue survives a broker restart; publishing with
     * the persistent delivery mode means the message does too. deadLetterExchange
     * is broker-native on RabbitMQ: the broker routes a rejected message itself,
     * with no cooperation from the consumer process.
     */
    @Bean
    Queue areaQueue() {
        return QueueBuilder.durable(QUEUE)
                .deadLetterExchange(DLX)
                .deadLetterRoutingKey(DLQ)
                .build();
    }

    /**
     * The half that is easy to forget: naming an exchange on the queue does not
     * create it. Nack toward an exchange that does not exist and RabbitMQ
     * discards the message silently: the exact loss this page is about,
     * reintroduced by an omission in configuration.
     */
    @Bean
    DirectExchange areaDlx() {
        return ExchangeBuilder.directExchange(DLX).durable(true).build();
    }

    @Bean
    Queue areaDlq() {
        return QueueBuilder.durable(DLQ).build();
    }

    @Bean
    Binding areaDlqBinding() {
        return BindingBuilder.bind(areaDlq()).to(areaDlx()).with(DLQ);
    }
}

@Component
@RequiredArgsConstructor
public class AreaChangedListener {

    private final AreaProjection projection;

    @RabbitListener(queues = AreaMessagingConfig.QUEUE, ackMode = "MANUAL")
    public void onAreaChanged(AreaChanged event,
                              Channel channel,
                              @Header(AmqpHeaders.DELIVERY_TAG) long tag) throws IOException {
        try {
            projection.apply(event);         // idempotent, see the next tab
            channel.basicAck(tag, false);    // the broker may now forget it
        } catch (TransientProjectionException e) {
            // requeue = true puts the delivery back at the head of the queue.
            // Reach for it only when you know the failure is transient AND you
            // have a delivery-count guard, or you have built a hot loop that
            // redelivers one poison message until someone notices the CPU.
            channel.basicNack(tag, false, true);
        } catch (Exception e) {
            // requeue = false hands it to the dead-letter exchange declared
            // above: preserved for inspection instead of dropped or looped.
            channel.basicNack(tag, false, false);
        }
    }
}

/**
 * At-least-once is only safe when applying the same event twice is
 * indistinguishable from applying it once. The deduplication claim and the
 * business write must commit or roll back together: a "seen" flag written
 * after the projection is not idempotence, it is a smaller race window.
 */
@Service
@RequiredArgsConstructor
public class AreaProjection {

    private static final String CONSUMER = "area-projection";

    private final ProcessedEventRepository processed;
    private final AreaReadModelRepository readModel;

    @Transactional
    public void apply(AreaChanged event) {
        if (processed.claim(event.eventId(), CONSUMER) == 0) {
            return;                       // an earlier delivery already applied it
        }
        readModel.upsert(AreaSummary.from(event));
    }
}

/** Composite primary key: one row per (event, consumer) pair. */
public record ProcessedEventId(String eventId, String consumer) implements Serializable {}

public interface ProcessedEventRepository
        extends JpaRepository<ProcessedEvent, ProcessedEventId> {

    /**
     * Returns 1 when this delivery claimed the event, 0 when a previous one did.
     *
     * ON CONFLICT DO NOTHING rather than catching DataIntegrityViolationException:
     * a constraint violation raised inside an active persistence context marks the
     * transaction rollback-only, so catching it and carrying on does not work.
     *
     * The id type must be the COMPOSITE key, not String. Declaring it as
     * JpaRepository<ProcessedEvent, String> would either fail to map or
     * deduplicate on event_id alone - which silently destroys the per-consumer
     * independence this table exists to provide, because the first consumer to
     * claim an event would block every other consumer from ever seeing it.
     *
     * Dialect: ON CONFLICT is PostgreSQL (and SQLite). On MySQL 8 use
     * INSERT IGNORE or INSERT ... ON DUPLICATE KEY UPDATE; SQL Server needs
     * MERGE or an existence check under the right isolation level.
     */
    @Modifying
    @Query(nativeQuery = true, value = """
            INSERT INTO processed_events (event_id, consumer, processed_at)
            VALUES (:eventId, :consumer, now())
            ON CONFLICT (event_id, consumer) DO NOTHING
            """)
    int claim(@Param("eventId") String eventId, @Param("consumer") String consumer);
}

/** The producer stamps the id. A broker-assigned id changes on redelivery. */
public record AreaChanged(String eventId, String missionId, String areaId,
                          String name, Instant occurredAt) {}

Dead-lettering deserves the same precision. It is not error handling; it is the decision to stop retrying and preserve the evidence, which is the pattern Enterprise Integration Patterns describes as what happens "when a messaging system determines that it cannot or should not deliver a message". A dead-letter queue with no alert on its depth is a silent data-loss channel with extra steps.

When Redis pub/sub is the right call

None of the above makes Redis pub/sub a bad tool. It is an excellent one, and the operational argument is strong: it is already running for caching and sessions, it adds no broker to provision or patch, and it delivers with in-memory latency because there is no disk in the path. That last property is not a shortcut around durability. It is the same decision, viewed from the other side.

• Live cursors, presence and typing indicators. A dropped position is corrected by the next one, roughly 50 ms later. Persisting these would be a bug, not a feature.
• Ephemeral UI fan-out. Toasts, "someone else is editing this", live dashboard ticks: state the user can re-derive by refreshing.
• Cache invalidation hints. The worst case of a dropped hint is a stale entry until TTL, which is the failure mode the cache already tolerates by design.
• Notify-and-reconcile. Write the durable record first, then publish a hint that says only "area 41 changed". Receivers that hear it refresh early; receivers that missed it refresh on their next poll. This is the single highest-leverage pattern on this page: it keeps pub/sub's latency and moves the durability requirement to a store that already has it.
• Anything the receiver's state is derived from: projections, counters, audit trails, outbound integrations.
• Anything a human is waiting on. "The operator did not see the geofence alert" is not recoverable by a later message.
• Anything you would need to replay to rebuild a downstream system after an incident.

When a channel crosses from the top group into the bottom group, the first move is usually not a new broker. Redis already ships the durable primitive:

Streams give you consumer groups, per-consumer pending entries, XACK, replay from any entry ID, and, as the pub/sub documentation itself points out, persistence, "both at-most-once as well as at-least-once delivery semantics". Migrating a channel from PUBLISH to XADD is a change of two call sites and a consumer loop. Adopting Kafka is a change of platform, staffing and on-call. Those are not comparable decisions, and reaching for the second when the first would do is how estates acquire infrastructure nobody can operate.

Canonical sources

• Redis: Pub/Sub · the at-most-once statement and the ordering guarantee, in the vendor's own words.
• Redis: Streams · the durable alternative that does not require a second piece of infrastructure.
• Apache Kafka Design: Message Delivery Semantics · the three definitions, the default guarantee, and the warning about exactly-once claims.
• Enterprise Integration Patterns: Guaranteed Delivery · the pattern name for "survives the messaging system crashing".
• Enterprise Integration Patterns: Publish-Subscribe Channel · one copy per receiver, and the Durable Subscriber variant that pub/sub alone does not give you.
• Enterprise Integration Patterns: Dead Letter Channel · where a message goes when the system decides it cannot or should not deliver it.
• Martin Fowler: What do you mean by "Event-Driven"? · why the term needs qualifying before it carries any information.