Breaking Indexing for Thousands of Customers with One Bug

on YouTube ↗24m

" summary

A postmortem of a production incident where a single bug caused indexing to fail across 10 shared clusters serving thousands of customers. It describes the root cause, the impact on indexing, and the hands-on recovery efforts and steps taken over several hours to restore the service.

AI-written summary · about

" chapters (13)

1. 0:00Introduction
2. 1:10Search engine architecture
3. 3:002016 critical bug
4. 4:29Black Friday deployment
5. 5:57Immediate crash impact
6. 7:20Crisis collaboration
7. 8:44Root cause
8. 10:14Emergency fix
9. 11:20Technical lessons
10. 13:54Deployment strategy
11. 15:00Crisis management
12. 16:46Post-mortem and actions
13. 20:01Q&A

" related talks

• 42mPostgres: stories that will scare youJérémie Grauer · PG Day France 2024
• 47m30 Indexes on a 6TB Postgres Table: Challenges and SolutionsFlora Briand, Emmanuel Quincerot · Devoxx France 2023
• 53mIndexing Internals for Developers & DBAsBradley Ball · SQLBits 2024
• 32mWhen product meets real world data - how to analyze and tune deployments & upgradesDominik Süß, Tom Blackford · adaptTo() 2018
• 24mClosing Remarks & Mastering Efficiency: Pathbreaking OpenSearch Index Management (Ingestion, Index)Eli Fisher, Srinivas Reddy · OpenSearchCon India 2024
• 15mDealing with Bad Architecture Decisions: A Case StudyClaire Janke · StarCon 2018
• 44mBlack Monday: The Story of 5.5 Hours of DowntimeDmytro Dziubenko · Highload fwdays'24 conference
• 53mIndexing Internals for Developers & DBAsBradley Ball · SQLBits 2022