Hidden gems of WAL-G backup tool
summary
WAL-G is a CLI backup tool used at ~25,000 clusters scale with ~11 PB of data and ~10M req/sec that focuses on faster restores and low impact on live CPU/IO. It offers LSN-based deltas (configurable N deltas, default six) for much faster restores, reverse delta unpacking that is ~3x faster, parallel and aggressive archival modes, WAL-based prefaulting, throttling and a turbo override, ketchup send/receive to incrementally catch up lagging standbys, timeline and WAL integrity checks, and planned snapshot and sharded-cluster support; some features (summarization, deltas, PITR variants) remain off by default due to past corruption and need revalidation.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:28About the speaker
- 4:36Usual backup features
- 14:43Parallel WAL archiving
- 18:34Delta files and prefault
- 20:02Partial restore limitations
- 21:29Throttling and turbo mode
- 22:55Consistency checks
- 26:30Reverse Delta restoration
- 27:16Ketchup for standby lag
- 28:44Future features
- 31:15Sharded cluster backups
- 31:57Snapshot backups
- 33:00Failover storage
- 34:49Backup and vacuum integration
- 35:11Project downsides
- 36:35Q&A session
related talks
49mCreating Backups for Various Transactional DBMSs Under Load
49mWhat can't pgBackRest do for you?
26mThe Wonderful World of WAL
18mIncremental Backup in PostgreSQL
31mPostgreSQL WAL and Commit Synchronization, Optimization Opportunities
51mHeavy Duty Backup w/PgBackRest
52mOn WAL Redundancy in Postgres
15mScaling PostgreSQL Change Data Capture Using Open Source Airbyte