Keynote – Jay Kreps
summary
Explains a concrete system: Apache Kafka is a stand alone commit log that feeds databases, caches, and real time processors and scales across data centers. It is built as a distributed service with partitioning, replication, and order guarantees, storing terabytes per server and delivering hundreds of MBps per node. Production numbers cited include over a trillion messages daily and roughly a petabyte retained while supporting thousands of engineers. Work includes connectors, notably a prototype connector for Postgres logical change capture.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:35Data Systems
- 3:20Adhoc Systems
- 3:50Data Flow Challenges
- 5:00Central Commit Log
- 6:10Kafka
- 6:45Log abstraction
- 9:30Transformation
- 11:00For Data
related talks
53mDemystifying Kafka
29mApache Kafka and the Next 700 Stream Processing Systems
39mKafka as a Platform: The Ecosystem from the Ground Up
41mKafka as a Platform: The Ecosystem from the Ground Up
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
17mOvercome By Events - Apache Kafka And Event-oriented Architecture
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
45mApache Kafka as the Single Source of Truth: Usage, Problems, Pain and Consequences