Debezium change data capture: Real‑Time Pipelines with Kafka, Streams, and the Outbox Pattern

Debezium streams database WAL events into Kafka, letting microservices react instantly.

Combined with the outbox pattern, it guarantees exactly‑once delivery across service boundaries.

Debezium change data capture provides sub‑second, exactly‑once data pipelines by streaming WAL events to Kafka.

Debezium change data capture – Introduction

Real‑time inventory updates and order state sync have become non‑negotiable after Google’s AI‑driven shopping test on Flipkart.

Insurers losing 942 M because stale data feeds inaccurate risk models illustrate the cost of lag.

At the same time, Crusoe’s 1.25 B AI data‑center pull‑back shows the industry’s appetite for energy‑efficient pipelines.

In this context, a CDC‑first architecture solves three problems:

1. Latency – sub‑second propagation from OLTP to downstream services.
2. Consistency – exactly‑once semantics despite retries, crashes, or network partitions.
3. Scalability – horizontal growth on Kubernetes without manual sharding. Debezium change data capture taps the transaction log of MySQL, PostgreSQL, or MongoDB, converting each commit into a Kafka record.

Kafka retains the order, provides durable storage, and lets Kafka Streams enrich or filter events before they reach consumers.

The outbox pattern sits on the producer side: business services write domain events to an “outbox” table in the same transaction that updates their core tables. Debezium then treats the outbox as just another source, guaranteeing atomicity between state change and event emission.

Why traditional approaches fall short

• Polling introduces seconds‑to‑minutes of drift and adds load on the source DB.
• Application‑level publishing risks lost events when the DB commit succeeds but the message send fails.
• Dual‑writes break atomicity, leading to eventual consistency bugs that are hard to debug.

Desired guarantees

What is Debezium change data capture?

Debezium is an open‑source connector suite for Apache Kafka that reads database change logs (WAL, redo logs, etc.) and emits a stream of change events.

Each event contains:

• before and after snapshots of the row.
• source metadata (transaction ID, timestamp, binlog position).
• op code (c for create, u for update, d for delete). The connector runs as a Kafka Connect worker, handling offsets, schema registration, and fault‑tolerant restarts automatically.

How does the outbox pattern complement CDC?

1. Transactional write – Service updates its domain tables and inserts a row into outbox_events in the same DB transaction.
2. Debezium captures – The outbox table appears as another source; its rows become Kafka messages.
3. Consumer reads – Downstream services consume from the outbox topic, process idempotently, and acknowledge. Because the outbox write shares the same transaction as the state change, there is no window where one succeeds and the other fails.

Problem Statement & System Architecture

Enterprises need a pipeline that:

• Propagates every OLTP change to any number of microservices.
• Guarantees exactly‑once semantics end‑to‑end.
• Scales to tens of thousands of TPS on Kubernetes.
• Offers observability for latency, lag, and schema drift.

Core components

1. Source DB – MySQL 8.x with GTID enabled.
2. Kafka Cluster – 3‑node KRaft mode, replication factor 3, min‑insync replicas = 2.
3. Kafka Connect – Debezium MySQL connector, distributed mode, offset storage in Kafka.
4. Outbox Table – outbox_events(id UUID PK, aggregate_id VARCHAR, type VARCHAR, payload JSONB, created_at TIMESTAMP).
5. Kafka Streams Application – Enriches events, applies business rules, writes to domain‑specific topics.
6. Consumer Services – Spring Boot, Quarkus, or Go microservices that read from their topic with exactly‑once processing enabled.
7. Observability Stack – Prometheus + Grafana dashboards for connector lag, stream processing throughput, and consumer lag.
8. Schema Registry – Confluent Schema Registry storing Avro schemas for all topics.

Data flow

Scaling strategy

• Connector scaling – Deploy multiple tasks; each task handles a subset of tables or partitions.
• Kafka Streams parallelism – Set num.stream.threads to match pod CPU limits; use stateless processors for linear scaling.
• Kubernetes autoscaling – Horizontal Pod Autoscaler (HPA) reacts to kafka.connect.task.max.poll.records and stream.processing.rate metrics from Prometheus.
• Back‑pressure handling – Enable consumer.max.poll.interval.ms and producer.delivery.timeout.ms to avoid OOM during spikes.

Architecture Patterns Comparison

About the Author

I'm a senior backend engineer with a focus on real‑time data pipelines and Kubernetes‑native deployments.

Feel free to reach out with questions or consulting inquiries: Contact Me.

Step-by-Step Implementation Guide

You've got the theory down. Now let's build the actual pipeline. We start with the database, move through Kafka, and end with the consumer that guarantees consistency.

1. Configure the Database for Logical Decoding

Debezium doesn't poll tables. It reads the Write-Ahead Log (WAL). You need to enable logical decoding in your PostgreSQL instance.

The wal_level = logical setting is non-negotiable. Physical replication uses a different format. Debezium needs the logical format to extract row-level changes.

Set max_replication_slots to match your expected number of Debezium instances. Too low, and you'll hit connection errors under load.

2. Deploy the Debezium Connector

Use the Kafka Connect REST API or a containerized deployment. Here’s a Docker Compose snippet for a single-node setup.

Register the connector via REST API. Note the transforms setting. We strip the schema to keep payloads lightweight.

The ExtractNewRecordState transform is critical. Without it, every message contains the full before/after state. That doubles your network bandwidth.

3. Implement the Outbox Pattern in Your Application

Don't write directly to Kafka from your service. Write to a local outbox table in the same transaction.

Production Pitfalls & Performance Optimization

Debezium change data capture works great until a hidden bottleneck surfaces. Below are the most common edge cases and how to tame them.

Tuning the Debezium Connector

• Keep tasks.max modest; more tasks increase parallelism but also raise MySQL connection count.
• max.batch.size controls how many rows Debezium pulls per poll. Larger batches reduce round‑trips but raise memory pressure.

Optimizing the Kafka Producer for Exactly‑Once

• enable.idempotence guarantees no duplicate writes.
• transactional.id lets the connector commit offsets atomically with the data payload.
• max.in.flight.requests.per.connection=5 balances throughput with ordering guarantees.

Outbox Batch Writer

• Group events in batches of 100–200.
• Use a single batchUpdate call to avoid per‑row round‑trips.
• The created_at timestamp lets downstream consumers dedupe safely.

Monitoring the Pipeline

• JVM metrics – Export jvm.memory.used and jvm.gc.pause to Prometheus.
• Kafka lag – Track consumer_lag per partition; alert when > 5000.
• Outbox backlog – Count rows where processed = false. If any metric crosses its threshold, automatically scale the connector task count or spin up an extra consumer group.

Final Summary & Key Takeaways

Debezium change data capture gives you a reliable, low‑latency feed from the source database. Pair it with Kafka’s exactly‑once semantics and an outbox table, and you have a robust event‑driven backbone. The trick is to keep each component tuned:

1. Connector configuration – Small batches, heartbeats, and explicit snapshot control prevent memory bloat.
2. Kafka producer settings – Idempotence and transactions lock down duplicates.
3. Outbox design – Composite keys, batch inserts, and a processed flag keep the write path fast and safe.
4. Observability – Export JVM, Kafka, and outbox metrics; set tight alerts. When you respect these boundaries, scaling is as simple as adding more connector tasks or consumer instances. The architecture remains linear, and you retain exactly‑once guarantees without sacrificing throughput.

How do I handle schema changes without breaking the pipeline?

Debezium stores schema evolution in a dedicated Kafka topic (schema-changes.*). Let the connector run in snapshot.mode=when_needed so it captures the new DDL once. Downstream consumers should read the schema topic and apply the latest Avro/Protobuf definition before deserializing events. If you need an immediate cut‑over, pause the connector, apply the DDL, then resume; the connector will emit a DDL event that downstream services can react to.

What’s the safest way to achieve exactly‑once when the outbox table is shared across services?

Wrap the outbox insert and the business transaction in a single DB transaction. After commit, let Debezium capture the row and forward it to Kafka. On the consumer side, enable enable.idempotence and use a transactional consumer (isolation.level=read_committed). The consumer should mark the outbox row as processed = true within the same transaction that publishes the downstream event, guaranteeing atomicity.

When should I consider moving from a single‑node Kafka cluster to a multi‑broker setup?

If you see any of these signals: sustained producer latency > 100 ms, consumer lag consistently above 10 k messages, or disk usage on the broker exceeding 70 %. Multi‑broker clusters also protect against node failure and let you increase partition count for parallelism. Start with three brokers for quorum, then scale partitions per topic to match your consumer parallelism.

If you’re ready to turn these concepts into production‑grade code, let’s talk. Manish Joshi blends deep Flutter UI expertise, AI‑augmented workflows, and battle‑tested FastAPI/Node.js back‑ends. He can help you design a real‑time CDC pipeline that scales, stays consistent, and integrates cleanly with your existing stack.

Get in touch with Manish Joshi →