What Is Change Data Capture (CDC)?

What is change data capture?

CDC emerged as an alternative to batch extraction. Traditional ETL pipelines re-query source tables on a schedule — hourly or nightly — which is slow, load-intensive, and blind to anything that happened between runs. CDC inverts the model: rather than asking "what is the state now?", it observes every change as it happens and propagates it continuously.

There are three main approaches. Log-based CDC reads the database's own transaction log — PostgreSQL's write-ahead log, MySQL's binlog, MongoDB's oplog. It adds minimal load to the source, captures every change including deletes, and preserves commit order; tools such as Debezium made this approach the modern default. Trigger-based CDC installs database triggers that copy changes into shadow tables. It works on databases without accessible logs but adds synchronous overhead to every source transaction. Query-based CDC polls tables for rows with a newer updated_at timestamp. It is the simplest to build but misses deletes, misses intermediate states between polls, and adds recurring query load.

How change data capture works

Log-based CDC in PostgreSQL is representative. The database is configured for logical decoding, a publication declares which tables to capture, and a connector creates a replication slot from which it streams decoded changes:

-- postgresql.conf: wal_level = logical
CREATE PUBLICATION cdc_pub FOR TABLE orders, payments;
-- A connector (e.g., Debezium) then creates a replication slot
-- and streams ordered change events decoded from the WAL.

Each change event carries the operation type, the before and after row images, and a log position for ordering. A new pipeline typically runs in two phases: an initial snapshot of existing rows, then continuous tailing of the log from the snapshot's position. Events are usually delivered through a message broker such as Kafka, partitioned by primary key so that changes to the same row stay in order.

Delivery is generally at-least-once: after a failure, a connector resumes from its last recorded position and may re-emit events it already sent. Consumers therefore need to be idempotent — applying the same change event twice must produce the same result as applying it once (see the CDC ingestion guide for a concrete consumer-side setup).

Why change data capture matters in real-time systems

CDC is the connective tissue of most real-time architectures: it is how caches stay warm, how search indexes track the catalog, and how analytical stores and derived views receive transactional changes without burdening the system of record.

The essential property to understand is that CDC propagation is asynchronous. A downstream consumer always lags the source by some interval — commonly sub-second in steady state, but stretching to seconds or longer during traffic bursts, initial snapshots, connector restarts, or downstream backpressure. A CDC-fed view is therefore a faithful picture of the recent past, not the present. For automated decisions reading that view, the replication lag is part of the freshness budget: a decision made at time T reflects source state as of T minus the lag, and the lag is largest exactly when event volume spikes. Measuring end-to-end lag, rather than assuming it, is the difference between a known and an unknown staleness bound.

CDC also solves a correctness problem: the dual-write trap. An application that writes to its database and separately publishes an event to a broker can fail between the two, leaving the systems permanently inconsistent. With CDC, the database commit is the single source of events — downstream systems derive from the same ordered log, so they converge on the same history.

FAQ