The Outbox Pattern with DynamoDB

Introduction

One of the most deceptively hard problems in distributed systems is this: how do you update your database and publish an event about that update, atomically, when the database and the message broker are two completely different systems with no shared transaction?

Get this wrong and you end up with one of two failure modes: you write to the database but the event never gets published (a downstream service silently drifts out of sync), or you publish the event but the database write fails or rolls back (consumers react to something that never actually happened). The Outbox Pattern solves this by turning a distributed consistency problem into a single-system, transactional one — and DynamoDB, with its support for transactional writes and native Streams, turns out to be a particularly good fit for implementing it.

The Dual-Write Problem

Consider a typical scenario: a payment service processes a transaction and needs to notify other services (fraud detection, notifications, ledger reconciliation) that the payment was completed. The naive approach looks like this:

1. Write payment record to database
2. Publish "PaymentCompleted" event to SNS/SQS/Kafka

This looks reasonable until you ask: what happens if step 1 succeeds and step 2 fails because of a network blip, a broker outage, or the process crashing between the two calls? There is no atomic operation spanning a database and a message broker — you cannot wrap both in a single transaction. This is the dual-write problem, and it is the root cause of a huge number of "why are these two systems out of sync" incidents in production.

How the Outbox Pattern Works

The Outbox Pattern sidesteps the dual-write problem by never actually doing a dual write. Instead:

1. When a business operation happens, the service writes both the business record and an "outbox event" record in a single atomic transaction, to the same database.
2. A separate process — a poller or a change-data-capture (CDC) mechanism — reads new outbox records and publishes them to the actual message broker (SNS, SQS, EventBridge, Kafka, etc.).
3. Once successfully published, the outbox record is marked as processed (or deleted).

Because the business write and the outbox write happen in the same transaction, they either both succeed or both fail — there is no window where one happened without the other. The publishing step becomes an "at least once" delivery problem, which is a much easier problem to solve reliably (via retries and idempotent consumers) than atomic cross-system writes.

Why DynamoDB Is a Strong Fit

DynamoDB has two features that make it particularly well-suited to implementing the Outbox Pattern cleanly, without needing a separate relational database just for this purpose.

1. TransactWriteItems

DynamoDB's TransactWriteItems API lets you group up to 100 write operations, across up to 100 different items (potentially across multiple tables), into a single all-or-nothing transaction. This is exactly the primitive the Outbox Pattern needs: you can write your business item and your outbox event item atomically, even if they live in different tables.

{
  "TransactItems": [
    {
      "Put": {
        "TableName": "Payments",
        "Item": {
          "PK": { "S": "PAYMENT#123" },
          "SK": { "S": "METADATA" },
          "status": { "S": "COMPLETED" },
          "amount": { "N": "150.00" }
        }
      }
    },
    {
      "Put": {
        "TableName": "OutboxEvents",
        "Item": {
          "PK": { "S": "OUTBOX#123" },
          "SK": { "S": "EVENT#2026-08-14T12:00:00Z" },
          "eventType": { "S": "PaymentCompleted" },
          "payload": { "S": "{\"paymentId\":\"123\",\"amount\":150.00}" },
          "status": { "S": "PENDING" }
        }
      }
    }
  ]
}

If either write fails — say, a conditional check on the payment item fails because it was already processed — the entire transaction rolls back, and no orphaned outbox event is left behind.

2. DynamoDB Streams as the Relay Mechanism

Rather than polling the outbox table on a schedule (which adds latency and read costs), DynamoDB Streams can capture every item-level change in near real time and forward it to a Lambda function. This gives you a natural, low-latency relay:

DynamoDB (outbox table) → DynamoDB Streams → Lambda → EventBridge / SNS / SQS

The Lambda function reads the stream record, extracts the event payload, publishes it to the actual broker, and — critically — must handle the possibility of duplicate delivery, since Streams-triggered Lambdas guarantee at-least-once processing, not exactly-once.

Reference Architecture

A typical outbox implementation on AWS looks like this:

• Business table — holds the actual domain entities (payments, orders, accounts).
• Outbox table — holds pending events, keyed so that stream processing and cleanup are efficient (e.g., partition key per aggregate, sort key with a timestamp or ULID for ordering).
• DynamoDB Streams — enabled on the outbox table, configured with NEW_IMAGE view type so the Lambda receives the full event payload.
• Lambda consumer — reads stream batches, publishes each event to EventBridge/SNS/SQS, and handles retries with exponential backoff.
• TTL on the outbox table — once an event has been successfully published, there's no need to keep it around. A TTL attribute lets DynamoDB automatically expire processed records instead of requiring an explicit delete, keeping the table lean.

Handling Idempotency on the Consumer Side

Because the relay step is at-least-once, not exactly-once, downstream consumers of the published events must be idempotent. Two common strategies:

• Idempotency keys — every outbox event carries a stable, unique identifier (e.g., a ULID generated at write time). Consumers track which IDs they've already processed and skip duplicates.
• Natural idempotency — design event handlers so that processing the same event twice produces the same end state (e.g., an upsert instead of an insert, or a conditional update based on version number).

This is a small amount of extra design discipline that buys you a huge amount of reliability — it's the same trade-off used in idempotent payment APIs, and it pairs naturally with a background in payment validation and processing pipelines.

Ordering Considerations

DynamoDB Streams preserves order within a single partition key, but not globally across partition keys. If strict ordering matters for a given aggregate (e.g., all events for a specific order must be processed in sequence), design the outbox table's partition key around the aggregate ID, so that all of its events land in the same shard and are streamed in order. Cross-aggregate ordering, if needed at all, generally has to be handled at the consumer level rather than relied upon from the stream itself.

Trade-offs and Failure Modes

No pattern is free. A few things worth planning for explicitly:

• Increased write cost — every business write now involves at least one additional item write inside the same transaction, plus the read/write cost of Streams and the relay Lambda.
• Relay lag — there's a small delay between the transactional write and the event actually reaching the broker, since it depends on Streams propagation and Lambda invocation. Most workloads see this in the low hundreds of milliseconds, but it's not zero, and shouldn't be treated as such for latency-sensitive flows.
• Poison messages — a malformed or unprocessable outbox event can block a shard's stream processing if not handled carefully. A dead-letter queue (DLQ) on the Lambda's event source mapping is essential, not optional.
• Monitoring the gap — you need visibility into how many outbox events are pending and how old the oldest unprocessed one is; otherwise a stuck relay can silently accumulate a large backlog before anyone notices.

When (Not) to Use It

The Outbox Pattern earns its complexity when downstream systems genuinely need to know about state changes reliably — notifications, analytics pipelines, other microservices reacting to domain events, or audit/compliance trails. It's overkill for purely internal reads within the same service, and it shouldn't be reached for by default whenever a message needs to be sent; if occasional message loss is truly tolerable for a given use case, a simpler fire-and-forget publish may be perfectly adequate.

Conclusion

The dual-write problem is one of those issues that looks small in a design doc and becomes a production incident months later, usually surfacing as "why does service B think this order is still pending when service A already completed it." The Outbox Pattern closes that gap by making the event publication part of the same atomic write as the business change, and DynamoDB's TransactWriteItems plus Streams give you a genuinely clean, serverless-native way to implement it — no extra message queue infrastructure required just to guarantee the write happened, and no separate relational database needed purely to get transactional semantics. For teams already running DynamoDB-backed microservices on AWS, it's one of the highest-leverage reliability patterns available, and one well worth having in place before, not after, the first inconsistency incident.