Druid Lookups for High Cardinality Dimensions

on YouTube ↗15m

" summary

Describes limitations of Druid’s built-in map and namespaced lookups for large, high-cardinality dimensions and explains a custom lookup framework that adds scalable, multi-column, real-time-capable lookups. The solution uses an embeddable persistent key‑value store for lookup storage, Kafka for messaging, and HDFS for deep storage to handle hundreds of millions of rows and support updates.

AI-written summary · about

" chapters (12)

1. 0:00Start
2. 1:23What's Druid?
3. 4:03Druid req/resp
4. 4:42What about mutable dimension data ?
5. 5:11Few options
6. 6:11What's druid lookups?
7. 6:55Limitations of default implementation
8. 8:03Introducing maha druid lookups
9. 11:50Salient features
10. 13:16Why RocksDB?
11. 14:23Integration with maha
12. 14:53Contributors

" related talks

• 45mDruid: Powering Interactive Data Applications at ScaleFangjin Yang · Strange Loop 2016
• 42mCasting the spell: Druid in practiceItai Yaffe, Yakir Buskilla · ApacheCon@Home 2021
• 27mCasting the Spell: Druid in PracticeItai Yaffe, Yakir Buskilla · Berlin Buzzwords 2021
• 39mApache Druid 101Matt Sarrel · Data Con LA 2020
• 45mHow We Got Druid to Work at OdnoklassnikiYuriy Nevinitsin · HighLoad++ 2017
• 28mCasting the spell: Druid in practiceItai Yaffe, Yakir Buskilla · Scale By the Bay 2021
• 46mScalable Realtime Analytics using DruidNishant Bangarwa · The Fifth Elephant 2016
• 40mBuild Real-time Analytic Applications: The Easy WaySergio Ferragut · Berlin Buzzwords 2022