Realtime Analytics Unleashed: Unlocking the Potential of Data at Scale

on YouTube ↗29m

" summary

Apache Pinot powers OLAP at OLTP speeds using Kafka for ingestion, Kubernetes on Azure for deployment, and StarTree’s star-tree index to do selective cubing so common query paths are pre-aggregated. LinkedIn goes from 1,000 Elasticsearch nodes serving ~1,500 qps to 75 nodes 5,000 qps for 70M+ users. Features: H3 geospatial, JSON and vector/text indexes, separate batch and streaming ingestion servers to avoid query impact, Docker demo shows millisecond latencies.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 1:30Why Realtime Analytics
3. 4:58LinkedIn
4. 8:01Distributed Data Store
5. 9:27Kafka
6. 10:22Elastic Search
7. 16:59Architecture
8. 18:53Who uses StarTree
9. 21:45Demo
10. 24:06Contact Barkha
11. 25:21StarTree levels

" related talks

• 46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and PinotK Wolok, T Berglund · Voxxed Days Zürich 2022
• 49mIntroduction to Real-Time Analytics with Apache PinotTim Berglund · GOTO Chicago 2023
• 44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOTNeha Pawar · Crunch Conference 2021
• 47mReal-time analytics for your users with Apache PinotF. Teychene · DevOps D-Day 2022
• 33mOptimizing Speed and Scale of Real-Time Analytics Using Apache Pulsar and Apache P...Mary Grygleski · Reactive Summit 2022
• 47mBuilding a Real-Time Analytics DatabaseTim Berglund · GOTO Chicago 2023
• 38mReal-Time Analytics: Going Beyond Stream Processing with Apache PinotKarin Wolok, Rong Rong · Open AI + Data Forum @ Open Source Summit North America 2022
• 34mReal-Time Analytics: Going Beyond Stream Processing with Apache PinotMark Needham, Karin Wolok · Open Source Summit Latin America 2022: Open AI + Data Forum