Keynote | Stream Processing at LinkedIn: Then and Now
summary
LinkedIn runs stateful stream processing at massive scale: Kafka ingests over 11 trillion incoming messages per day and stream jobs process over 4 trillion messages per day. Title-standardization and notification pipelines migrate from a pure Kappa to a lambda-redux, using Apache Beam and SQL to share APIs, moving RocksDB state files to object storage (a rocksdb cloud pattern) to cut restart time and cost, and adding nearline learners to refine offline ad models while grappling with repartitioning, time boundaries and autoscaling.
AI-written summary · about
chapters (12)
- 0:00Linkedin stream processing
- 2:32Evolution of use cases
- 4:10Job title standardization
- 6:19Kappa architecture approach
- 9:46Lambda redux and efficiency
- 11:53Lingering challenges
- 13:24Member notification system
- 16:07RocksDB and blob storage
- 19:59Scalability and load
- 20:42AI in ad relevance
- 25:55Agile machine learning
- 28:51Future data challenges
related talks
52mKeynote Interview
37mStreaming your shared ride
01h 47mRadanalytics.io streaming applications workshop
18mThe role Stream Processing plays for an Opti Channel Experience
19mAI Workflows with Kafka and Modern Architectures
01h 02mBig Data Analytics in Near-Real-Time with Apache Kafka Streams
02h 16mKeynote Day 2 - Full
21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra