High-concurrency distributed snapshots

on YouTube ↗51m

" summary

Current MVCC snapshots rely on a shared-memory proc array and get_snapshot_data that becomes an N² scalability bottleneck as CPUs and concurrent transactions grow; subtransaction tracking is limited to 64 entries and forces expensive top-level lookups, and mixing synchronous_commit modes creates different commit orders on primary and standby and even lost commits on failover. Proposal: assign a monotonic commit sequence number (CSN) (counter, LSN, or hybrid wall-clock) recorded in an SLRU or ring buffer, batch-convert old CSNs to pg_xact bits, and use CSN-based snapshot checks — patches exist aiming for integration around Postgres 18.

AI-written summary · about

" chapters (5)

1. 0:00Introduction
2. 1:00Database concurrency concepts
3. 11:11Current snapshot implementation
4. 24:17Proposal for CSN snapshots
5. 43:22Optimization and future work

" related talks

• 48mLightning TalksLukas Fittl, Chioma Onyekpere, Dian Fay, Alastair Turner, James Miller, Stacey Haysler, Grant Zhou, David Wein, Akash Shankaran, Rintaro Ikeda, Ruohung Feng · PGConf.dev 2025
• 27mImplementing Strict Serializability with pg_xactJimmy Zelinskie · POSETTE: An Event for Postgres 2025
• 49mConcurrency Deep-DiveTejas Manohar · PostgresOpen Silicon Valley 2017
• 47mA Batch of Commit BatchingPeter V Geoghegan · Postgres Open 2012
• 45mTransactions and Concurrency Control PatternsVlad Mihalcea · Voxxed Days Zurich 2017
• 18mAvoiding Conflicts in Highly-Concurrent ApplicationsAlec Grieser · FoundationDB Summit 2018
• 28mUnderstanding Transaction Isolation for App DevelopersChandra Sekhar Pathivada · PGConf NYC 2022
• 41mTransactions: myths, surprises and opportunitiesMartin Kleppmann · Strange Loop 2015