ElasticSearch and Heka: How We Learned to Sieve an Elephant Through a Sieve

on YouTube ↗54m

" summary

Description of a real-time log collection system built at Yandex.Money that uses ElasticSearch and Heka and currently processes about 2 billion messages per day. Explains why the EHK (ElasticSearch + Heka) stack was chosen and details how the system was designed and implemented.

AI-written summary · about

" chapters (6)

1. 0:00Introduction
2. 6:46Log collection problems
3. 15:38Heka and ElasticSearch choice
4. 24:56Monitoring and security
5. 34:19Summary and recommendations
6. 35:26Q&A

" related talks

• 16mWhy we built ElasticSearchShay Banon · dotScale 2013
• 40mElasticSearch: Big Data, Search and AnalyticsShay Banon · Berlin Buzzwords 2012
• 42mGoing ElasticPhilipp Krenn · Codemotion Amsterdam 2016
• 01h 00mELK Stack - All You Need for Search In Your ApplicationsRohit Bhardwaj · Developers Summit 2021
• 01h 00mElasticsearch: you know, for more than searchRuss Cam · NDC Sydney 2017
• 35mElasticsearch - Beyond Full-text SearchAlex Reelsen · GOTO Aarhus 2013
• 44mElasticsearch at Scale of Billions of DocumentsIgor Kupczyński · Confitura 2015
• 24mOperating at scale with Elastic Search in productionSubramaniajeeva Kandasamy · ODSC India 2018