Infinite* Topic Backlogs with Apache Pulsar

on YouTube ↗46m

" summary

Claim: Apache Pulsar scales topic backlogs effectively to “infinite” size by separating serving and storage and using BookKeeper ledgers broken into immutable segments. Older closed segments are offloaded to tiered storage (S3/GCS/HDFS) with an index for random-access; open segments stay on fast storage. Demo shows a ~10 GB backlog and a stream of ~50 msgs/sec feeding a min‑count sketch function; Pulsar SQL (Presto) queries backlogs and tiered storage is production since v2.1/v2.2.

AI-written summary · about

" chapters (13)

1. 0:00Introduction
2. 4:17Lambda Architecture
3. 6:53Pulsar Introduction
4. 9:31Topic Backlogs
5. 11:31Segments
6. 16:12Storage Nodes
7. 18:07Mirroring
8. 20:42Tiered Storage
9. 23:46Indexing
10. 25:43Demo
11. 33:16Pulsar SQL
12. 34:31Pulsar Deployment Model
13. 39:26Wrap Up

" related talks

• 51mApache Pulsar 101: architecture, concepts & benchmarksQuentin Adam, Steven Le Roux · Devoxx Belgium 2019
• 55mEvent Messaging and Streaming with Apache PulsarMary Grygleski · Great International Developer Summit 2023
• 45mApache Pulsar: Finally an Alternative to Kafka?Julien Jakubowski · Devoxx France 2024
• 34mEvent Streaming in the Cloud Native World with Apache PulsarMary Grygleski · Cloud Builders: Java Conf 2023
• 14mFast Data on-Ramp with Apache Pulsar on K8Timothy J Spann · Kubernetes Batch + HPC Day Europe 2022
• 35mStreaming Data Made EasyTimothy J Spann, David Kjerrumgaard · OSA Con 2022
• 48mApache Pulsar: Event Streaming in The Cloud Native WorldMary Grygleski · GOTO Chicago 2023
• 33mOptimizing Speed and Scale of Real-Time Analytics Using Apache Pulsar and Apache P...Mary Grygleski · Reactive Summit 2022