Change data capture with Debezium and Kafka: what I learned running it in production

I use Debezium in two places on a logistics SaaS platform. One streams changes from a legacy SQL Server into the newer PostgreSQL services. The other relays an outbox table from PostgreSQL to Kafka. Both work, and both taught me things the docs mention only in passing. This post is what I would tell myself before starting.

When CDC is worth it

Reach for CDC when two systems must agree and you cannot change the writer much, or when you cannot afford to lose an event. The legacy SQL Server was the first case: nobody wanted to touch its code, but the new services needed its data within seconds. A cron job that polls for changes was already running as a safety net, and it still is, but it can only be as fresh as its interval.

The second case is the dual write. A service saves a row, then calls Kafka. That is two commits on two systems with no shared transaction. If the process dies in between, you lose the event. If you publish first and the database rolls back, you send an event for something that never happened. We had exactly the first bug: users who existed in some services and not in others, because the publish after commit failed silently.

If you have no Kafka Connect cluster yet, a polling publisher on an outbox table is simpler. I chose CDC for the outbox because the Connect cluster already existed for the SQL Server pipeline.

Architecture

For the outbox, the service writes the business row and an outbox row in the same local transaction. Debezium tails the PostgreSQL WAL and its outbox event router publishes each row to a topic. Consumers read from Kafka.

service -> [business row + outbox row] one transaction
        -> WAL -> Debezium (outbox router) -> Kafka topic -> consumers

A trimmed connector config, with placeholders instead of real names:

{
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  "plugin.name": "pgoutput",
  "slot.name": "outbox_slot",
  "publication.name": "dbz_outbox",
  "table.include.list": "public.outbox",
  "snapshot.mode": "never",
  "heartbeat.interval.ms": "300000",
  "transforms": "outbox",
  "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
  "transforms.outbox.table.field.event.id": "event_id",
  "transforms.outbox.route.topic.replacement": "domain.events"
}

The event id column becomes a message header, the aggregate id becomes the message key, and the payload becomes the value. Those three choices matter later.

Traps

A forgotten slot fills the disk. A PostgreSQL replication slot keeps WAL until its consumer reads it. If Debezium is down, or a connector was deleted and the slot was not, WAL piles up until the disk is full, on a database shared by a dozen services. I set max_slot_wal_keep_size = 10GB so an orphaned slot gets invalidated instead of taking the database down. I also check pg_replication_slots for inactive slots. The heartbeat above lets the slot advance when the table is quiet.

Snapshots. snapshot.mode=never means rows that existed before the slot are never relayed. For an outbox that is fine, but I had written test rows into the table beforehand and they were silently skipped. Decide the mode on purpose, and never insert test rows into an outbox that routes to a real topic, because the consumers will process them.

SQL Server has its own failure modes. CDC depends on SQL Server Agent: if the capture job stops, Debezium stays connected and sends nothing. The cleanup job deletes change-table rows after a retention period, 3 days by default. If the connector lags past that, you need a new snapshot. You also cannot TRUNCATE a table with CDC enabled.

Connect’s internal topics. Auto-created offsets, configs and status topics took the broker defaults and were wrong. The symptom was a connector in RUNNING state with zero tasks and no error, while WAL kept growing. Create those topics yourself, with the right partition counts and cleanup.policy=compact, before first start.

Ordering. Kafka orders per partition only. Using the aggregate id as the key keeps all events for one entity in order. Events for different entities have no order between them, and consumers should not assume one. I would rather design handlers that tolerate that than chase global order.

Schema change. I have fewer scars here than the topic deserves, so take this as design intent. The outbox payload is its own contract, so renaming a column on a business table does not reach consumers. Raw table CDC, like the SQL Server pipeline, does expose the table schema, and there I rely on additive changes and deploying consumers before the producer. The deploy-order rule in our notes comes from the same idea: a producer that adds a field must not ship before the consumers that need it, or events land in the dead letter topic in bulk.

Idempotent consumers

The relay is at-least-once. Debezium can send a row and crash before recording its offset, so duplicates will happen. Kafka’s idempotent producer does not help here, because it covers the Kafka boundary and not your database.

Each consumer therefore records processed event ids in a table, in the same transaction as its business write:

INSERT INTO processed_event (consumer_id, event_id, processed_at)
VALUES (:consumer, :event, now())
ON CONFLICT DO NOTHING;
-- 0 rows inserted: duplicate, skip the business write

The key is the consumer group plus the event id from the header, so two consumers can handle the same event independently. Rows older than a week are cleaned by a cron job. Taking the id from the header, not from the payload, also meant I did not parse anything just to check for a duplicate.

Two smaller bugs are worth knowing. Writing the payload as a string into a JSONB column stored a doubly escaped string, and consumers decoded it wrongly. And a couple of event factory methods forgot to set the event id, which caused null pointer errors on the consumer side.

What I would do again

1. Start with the outbox, not table-level CDC, when you own the writing service.
2. Set max_slot_wal_keep_size and alert on slot state and lag before the first connector goes live. An unmonitored slot is the failure that hurts most, and it is cheap to prevent.
3. Keep a periodic batch sync next to CDC for the legacy source. It fills gaps after a connector restart.
4. Rehearse kill-and-replay: stop Debezium mid-stream, restart it, replay old offsets, and check that nothing is lost or duplicated.

CDC moves the hard part from “publish reliably” to “operate a replication slot and write consumers that tolerate repeats”. That is a better problem to have, but it is still a problem.