Apache Pulsar 101: architecture and features

on YouTube ↗50m

" summary

Apache Pulsar's design separates stateless brokers from durable storage using BookKeeper-backed ledgers, letting brokers scale independently while topics are virtual and segments migrate between bookies for fast recovery. The talk covers Pulsar's protocol handlers (Kafka, RabbitMQ, MQTT), schema registry (Avro/JSON/Protobuf), server-side deduplication, dead-letter/retry, tiered offload to S3, multi-cluster geo-replication, emerging transactions for Flink, and production/operational aspects including a Kafka protocol handler and Biscuit tokens for fine-grained authorization.

AI-written summary · about

" chapters (10)

1. 0:00Introduction
2. 1:04Project overview
3. 9:36Layered architecture
4. 16:15Subscription models
5. 22:58Schema and transactions
6. 29:56Performance and isolation
7. 35:50Storage and retention
8. 40:03Multi-cluster geo-replication
9. 43:14Integrations and tools
10. 45:25Q&A

" related talks

• 51mApache Pulsar 101: architecture, concepts & benchmarksQuentin Adam, Steven Le Roux · Devoxx Belgium 2019
• 45mApache Pulsar: Finally an Alternative to Kafka?Julien Jakubowski · Big Data & RPA Conference Europe 2024
• 45mScalable distributed messaging&streaming with Apache PulsarJulien Jakubowski · Berlin Buzzwords 2023
• 46mApache Pulsar: Finally an Alternative to Kafka?Julien Jakubowski · DevDays Europe 2024
• 45mApache Pulsar: Finally an Alternative to Kafka?Julien Jakubowski · Devoxx France 2024
• 28mBuild your next-generation stream platform based on Apache PulsarNeng Lu · Flink Forward Virtual 2020
• 46mScalable messaging and streaming with Apache PulsarJulien Jakubowski · Voxxed Days Luxembourg 2023
• 53mEvent Messaging and Streaming with Apache PulsarMary Grygleski · Build Stuff 2022