Optimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot

on YouTube ↗46m

" summary

Apache Kafka streams events while Apache Pinot serves low‑latency OLAP queries so product features can show fresh analytics to users rather than just dashboards. The stack ingests batch and streaming sources, consolidates into columnar segments, and uses pluggable indexes (notably the star tree index, plus geospatial and JSON/text indexes) to hit sub‑50ms query latency at scale — LinkedIn reports about 1M events/sec ingest and 200k queries/sec in production.

AI-written summary · about

" chapters (4)

1. 0:00Start
2. 26:05What is Apache Pinot?
3. 27:44Powered by Apache Pinot
4. 35:29Star-Tree Index

" related talks

• 44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOTNeha Pawar · Crunch Conference 2021
• 29mRealtime Analytics Unleashed: Unlocking the Potential of Data at ScaleBarkha Herman · Shift Miami 2023
• 49mIntroduction to Real-Time Analytics with Apache PinotTim Berglund · GOTO Chicago 2023
• 34mReal-Time Analytics: Going Beyond Stream Processing with Apache PinotMark Needham, Karin Wolok · Open Source Summit Latin America 2022
• 47mReal-time analytics for your users with Apache PinotF. Teychene · DevOps D-Day 2022
• 39mFaster Analytics for Fast Data with Apache Pinot and Flink SQLChinmay Soman · Berlin Buzzwords 2021
• 34mReal-Time Analytics: Going Beyond Stream Processing with Apache PinotMark Needham, Karin Wolok · Open Source Summit Latin America 2022: Open AI + Data Forum
• 54mAn Introduction to Apache PinotTim Berglund · Devnexus 2022