Skip to content
Sign in
Artwork for My Weird Prompts
My Weird Prompts · Tuesday · 21 min

Why Database Replication Is Not Archival

Database pipelines between Postgres and modern data warehouses are often assumed to be archival conveyors for aging records, but real-world setups replicate live data continuously. Modern change data capture relies on logical decoding directly from the write-ahead log using tools like pgoutput, Debezium, and managed zero-ETL options. Examining everything from snapshot mechanics and warehouse billing intervals to unacknowledged replication slots reveals how downstream consumers can quietly exhaust primary storage disks if unmonitored. Key sources: • Debezium PostgreSQL connector docs, stable version — https://debezium.io/documentation/reference/stable/connectors/postgresql.html • Snowflake Openflow Connector for PostgreSQL, GA — https://docs.snowflake.com/en/user-guide/data-integration/openflow/connectors/postgres/about • Google Cloud Datastream for BigQuery — https://cloud.google.com/datastream-for-bigquery Episode 6427 — myweirdprompts.com/n/6427 (permanent short link: myweirdprompts.com/351399) Listen on Android: MWP FM Player — myweirdprompts.com/apk

0:00 · Intro-21:42
Play in
Transcript file

transcript

No transcript — this publisher did not publish one.

show notes

Database pipelines between Postgres and modern data warehouses are often assumed to be archival conveyors for aging records, but real-world setups replicate live data continuously. Modern change data capture relies on logical decoding directly from the write-ahead log using tools like pgoutput, Debezium, and managed zero-ETL options. Examining everything from snapshot mechanics and warehouse billing intervals to unacknowledged replication slots reveals how downstream consumers can quietly exhaust primary storage disks if unmonitored.

Key sources:
• Debezium PostgreSQL connector docs, stable version — https://debezium.io/documentation/reference/stable/connectors/postgresql.html
• Snowflake Openflow Connector for PostgreSQL, GA — https://docs.snowflake.com/en/user-guide/data-integration/openflow/connectors/postgres/about
• Google Cloud Datastream for BigQuery — https://cloud.google.com/datastream-for-bigquery

Episode 6427 — myweirdprompts.com/n/6427 (permanent short link: myweirdprompts.com/351399)

Listen on Android: MWP FM Player — myweirdprompts.com/apk

chapters

7 chapters

more episodes

All episodes