ExpertDatabase & Storage

PostgreSQL Change Data Capture (CDC) with Debezium

Stream database row-level changes in real-time to microservices or data warehouses by reading the PostgreSQL Write-Ahead Log (WAL) via Debezium.

PostgreSQLDebeziumKafka

Prerequisites

PostgreSQL with wal_level=logical
Kafka cluster running

Configuration Files

connector.jsonAPI POST payload

{\n  "name": "inventory-connector",\n  "config": {\n    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",\n    "database.hostname": "postgres",\n    "database.user": "replicator",\n    "database.password": "secret",\n    "database.server.name": "dbserver1",\n    "plugin.name": "pgoutput"\n  }\n}

Explanation:Registers the Debezium Postgres connector using the native pgoutput logical decoding plugin.

Verification Steps

Reads the change events from the Kafka topic.

$kafka-console-consumer.sh --topic dbserver1.public.users

Expected Output{"payload":{"before":null,"after":{"id":1,"name":"Alice"}}}

Production Gotchas

If the Kafka connector stops, the Postgres replication slot will retain WAL files indefinitely, eventually filling up the database disk.

Frequently Asked Questions

What is a replication slot?

A mechanism that ensures the primary database does not delete WAL segments until they have been consumed by the replica.