Event Delivery & Failure Guarantees

System Invariant

A successfully accepted event must remain recoverable until every subscribed consumer has successfully acknowledged its delivery. Temporary network or consumer failures must not silently turn persisted work into a lost delivery.

At the same time, delivery to one consumer must not determine the delivery state of another. Each subscriber therefore has independent delivery state even when multiple services receive the same event.

System Context

The platform consisted of multiple microservices that needed to communicate asynchronously. Rather than having producers communicate directly with every consumer, an internal Event Management Service (EMS) acted as the intermediary.

EMS exposed two gRPC operations. Producers used a regularPublishRPC to submit events, while consumers opened a long-lived bidirectional streaming RPC through which events and acknowledgements could be exchanged.

A consumer established its subscription when the service started and listened to the event stream from a separate goroutine. EMS maintained the currently connected subscriber streams in memory so that persisted events could be pushed directly to connected services.

Why This Is Hard

A naïve implementation could simply forward an event over the gRPC stream and consider the operation complete. That approach breaks as soon as the process, network, or consumer fails at the wrong moment.

Consider an event that is successfully transmitted to a consumer, but whose acknowledgement is lost before reaching EMS. EMS cannot determine whether the consumer received the event or not. Treating the timeout as success risks losing the event; treating it as failure means the event may be delivered again.

The system therefore had to accept that duplicate transmission was possible and make that outcome safe through consumer-side deduplication.

Core Requirements

• Authenticate and validate event publishers before accepting events
• Verify that publishers are authorized to publish to a requested topic
• Route events only to services subscribed to the relevant topic
• Persist events and subscriber delivery state before relying on asynchronous delivery
• Support long-lived streaming connections for consumers
• Require acknowledgement before advancing ordered delivery
• Retry transmission failures, negative acknowledgements, and uncertain timeouts
• Preserve event ordering for each subscriber
• Prevent duplicate event persistence on the consumer
• Recover pending work after process failures

Publishing

Producers did not publish business events directly from the request path. Each service used a transactional outbox so that the business transaction and creation of the event could be persisted together.

A publisher worker read pending records from the service's outbox and synchronously invoked the EMSPublishRPC. If EMS was unavailable or rejected the request, the publisher received a gRPC error and the outbox record remained the producer's durable source of work for a later attempt.

EMS performed validation before accepting the event. In particular, it checked whether the publisher was known and whether the publisher had permission to publish to the requested topic.

Topic Subscriptions

Consumers subscribed when their service started. The subscription established a long-lived bidirectional gRPC stream with EMS.

EMS maintained an in-memory mapping between the service name and its active stream connection.

When an event was accepted for publication, EMS resolved the subscribers interested in that event's topic and created delivery work for each of them.

Subscriber Connection Lifecycle

Subscriber streams were long-lived, but not assumed to be permanent. Consumers had their own re-dial and reconnection mechanism. When EMS detected a timeout or broken stream, it removed the connection from its in-memory subscriber map and cancels the service's subscription context too. Persistent delivery records were not removed, so a reconnecting consumer could continue from durable work.

Kubernetes introduced a less obvious race during rolling deployments. A new pod could become ready while the old pod was still running. Both pods could attempt to subscribe, and the old pod could overwrite the new pod's stream in the EMS map just before Kubernetes terminated it. EMS would then hold a stale stream instead of the new pod's connection.

I used Kubernetes' HOSTNAMEto distinguish pod instances. The connection lifecycle could then identify which pod owned the active subscription instead of treating every connection from the same service name as interchangeable.

Durable Delivery

Once an event passed validation, EMS persisted the event and created delivery records associated with its subscribers.

each subscriber had its own delivery state. Service A succeeding did not cause the event to be considered delivered to Service B or Service C.

The persistent queue therefore represented durable work, while the active gRPC stream represented only the current transport path.

Dispatcher & Wake-Up Model

EMS had a goroutine responsible for reading pending delivery work from the database and forwarding it to connected subscribers.

Inserting delivery work into the database was followed by a signal through a Go channel. The channel was not the queue itself; it was simply a wake-up mechanism that allowed the dispatcher to react without continuously polling the database.

A timer also triggered the dispatcher periodically. This provided a recovery path for work that existed in persistent storage even when there was no immediate channel signal. The dispatcher could also reconsider records still marked as processing, which was part of the single-EMS recovery model.

The dispatcher therefore had two ways to discover work: an immediate signal after new work was inserted and periodic recovery through the timer. A thread-safe execution guard prevented overlapping dispatcher runs.

Per-Service Ordering

Delivery was serialized independently for each subscriber.

The dispatcher did not advance to the next event for a service until the current event had been successfully acknowledged.

This serialization was scoped to the subscriber. Different services could progress concurrently.

The ordering decision deliberately reduced throughput because one slow or failing event could prevent later events for the same service from progressing. The system accepted that cost in exchange for predictable per-service ordering and simpler failure semantics. The implementation ordered candidate work using topic weight and the publication timestamp supplied with the event.

Acknowledgements & Retry

After sending an event through the subscriber stream, EMS waited for a response before considering that delivery complete.

A successful acknowledgement allowed the delivery state to advance. A transmission failure or negative acknowledgement left the event eligible for another attempt. Retry timing used exponential backoff in the relevant retry paths.

Timeouts were treated as uncertain outcomes. EMS could not know whether the consumer had received the event but the acknowledgement was lost, or whether the event had never reached the consumer. The safe choice was therefore to retry.

Consumer Inbox & Deduplication

Because retries could result in duplicate transmission, consumers maintained an inbox and a longer-lived inbox log.

When an event arrived, the consumer checked whether its event ID had already been recorded in the inbox log. If it was new, the event payload and delivery metadata were persisted in the inbox and the event ID was retained in the inbox log for deduplication.

Once the event had been durably accepted by the consumer, the consumer sent a successful acknowledgement back to EMS.

The inbox was treated as short-lived processing state: once an event was processed, its inbox record could be cleared to keep the table small. The inbox log was retained much longer so that duplicate deliveries could still be recognized. This made duplicate delivery safe at the persistence boundary and allowed the broker to use at-least-once delivery semantics rather than claiming exactly-once transport.

Deployment Scope

EMS was intentionally deployed as a single instance. The system did not attempt distributed broker ownership, consumer leases, or multi-instance stream coordination. This was a deliberate scope decision: the in-memory stream registry and serialized dispatcher were designed around one EMS process.

If EMS was unavailable, producer publisher workers did not treat the event as successfully published. The event remained in the producer service's transactional outbox and could be retried when EMS became available again. This kept EMS availability from becoming a data-loss boundary for producers.

Failure Modes

EMS fails after persistence

If EMS persisted the event and delivery records but failed before the in-memory wake-up signal could cause delivery, the event remained in persistent storage. The periodic dispatcher invocation could discover the work again. The deployment assumed a single EMS instance, so the design did not use distributed delivery leases or multi-instance ownership.

ACK is lost

A consumer may persist an event and send an acknowledgement that never reaches EMS. EMS eventually times out and retries the event. The consumer recognizes the event ID from its inbox log and avoids persisting the event again.

Consumer handler fails

Persisting the event and acknowledging its delivery were deliberately separate from executing the business handler. The ACK meant only that the event had been durably received by the consumer. A separate goroutine consumed the inbox and invoked the business handler. If handler execution failed, the ordered processing path could remain blocked while the event was retried later.

Poison event

Strict ordering creates an important failure mode: a permanently invalid event can prevent later events for that subscriber from progressing. This was identified as a limitation of the design and a reason to consider bounded retries and dead-letter handling.

Tradeoffs Accepted

Reliability over throughput: serializing event delivery per subscriber reduced concurrency, but simplified ordering and made delivery progression deterministic.

At-least-once over exactly-once: uncertain network outcomes made duplicate transmission unavoidable. Consumer-side deduplication was used instead of trying to make the transport exactly once.

Database-backed durability: persistent delivery state was preferred over relying on in-memory queues. Go channels were used as wake-up signals rather than as the durable source of work.

Simple coordination: the design used a serialized execution path with a thread-safe execution guard to prevent overlapping dispatcher runs. This kept the implementation easier to reason about at the cost of additional throughput constraints.

What I'd Change

The first improvement I would make is to introduce explicit retry state and bounded retries for consumer-side processing. A permanently failing event should not be allowed to block an entire ordered stream indefinitely.

Failed events could eventually be moved to a dead-letter queue after a defined number of attempts, allowing subsequent events to progress while preserving the failed payload for investigation or replay. This was an identified improvement rather than a guarantee of the version described here.

I would replace publication timestamps as the fundamental ordering primitive with an explicit sequence or ordering key. The existing implementation ordered using topic weight and publication timestamp, while acknowledging that producer clocks can drift.

If throughput eventually became the dominant constraint, the next evolution would be partition-level ordering rather than serializing every event for an entire subscriber. That would preserve ordering where required while allowing independent partitions to progress in parallel.