How to Protect the SQL Engine From Running Out of Memory
summary
Claim: TiDB prevents SQL engine OOMs by measuring and capping per-query memory with a mem tracker tree and a chain of OOM actions. Implemented actions include spill-to-disk for pipeline breakers (hash join, external sort) and an adaptive rate-limit that suspends or drops producers for scans. Measured results: a TPC-H-like run drops peak from ~8 GB to ~1 GB under quota, and a 200 GB dump finishes with memory ~500 MB. The components are built and shown to work.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:09Agenda
- 2:15SQL engine runs out of memory
- 3:06Scenario summarizaion
- 4:12Solution
- 5:00Pipeline breaker
- 7:47Pipelining operators
- 12:02Interfaces
- 12:47DiskSpillAction
- 13:53Rate LimitAction
- 18:021. SQL engine builds a query plan tree. 2. Sort starts to work 3. HashJoin starts to work
- 19:13Future work
related talks
15mOLTP SQL Database Query Tracing and Linting
57mSQL Server hates you(?) - what the DBAs never told the developers
48mMastering TempDB
50mQuery Store Best Practices
48mPractical Memory Tuning for PostgreSQL
54mArchitecture of a Reliable In-MemorDBMS: the Tarantool Example
29mDistributed SQL How TiDB Modernized Data Management
29mDistributed SQL: How TiDB Modernized Data Management