How to lose 50 Million Records in 5 minutes
summary
A production ETL called FORGE stores roughly 60 million records in Elasticsearch but two config worlds (YAML vs dynamic mappings) let timestamp-like JSON field names proliferate under Elasticsearch’s dynamic mapping, bloating cluster state. An attempted ES upgrade and earlier manual index deletions triggers recreation with exploding schemas and the loss of about 50 million records; recovery happens by restoring backups and rebuilding FORGE 2.0 with local raw storage and reindex tools.
AI-written summary · about
chapters (10)
- 0:00Intro
- 1:38Real Estate
- 5:27extraction
- 6:45transformation
- 11:08status quo
- 12:19path to ruin
- 19:47how to handle catastrophe
- 21:02how to avoid catastrophe
- 22:53FORGE
- 24:37final thoughts
related talks
29mHow (not) to Scale Elasticsearch for Data Analytics!
44mSurviving Elasticsearch
01h 01mBest Practices With Elasticsearch
20mReindexing in record time (...)
41mTips and Tricks to Scale Elasticsearch for 1M RPMs and Beyond
47mMake sense of your (big) data using Elasticsearch
44mElasticsearch at Scale of Billions of Documents
54m7 Petabytes of Logs in Elastic. How We Did It