Designing Scalable and Extendable Data Pipeline for Call of Duty Games
summary
Call of Duty telemetry amounts to petabytes, many hundreds of Kafka topics and tens of thousands of messages per second with sizes from 200 bytes to 20KB, and topic names that explode combinatorially across titles, platforms and environments. The decisions are specific: replace metadata-heavy topic names with a namespace + data-type convention, add an operational stream layer called refinery on Kafka Streams to filter and route, wrap payloads in a unified message envelope (protobuf with byte[] payload and message ID) and use a schema registry (Python + Cassandra) for deserialization and governance.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:22Scaling the pipeline
- 2:50Scaling Kafka
- 4:30Topic naming convention
- 8:10New topic naming convention
- 10:12Stream processing layer
- 12:38Single message schema
- 13:40Message envelope
- 15:44Schema registry
related talks
19mZero to Scale: Telemetry pipeline with Apache Cassandra
35mKafka Deployment at Scale
57mLessons in Building a System that Processes More than 70 Billion Events Daily
49mLet’s Build a Streaming Data Pipeline!
40mReal-time Customer Engagement in Gaming Using Kafka and Flink
40mReal-time Customer Engagement in Gaming Using Kafka and Flink
40mThe Changing Face Of ETL: Event-driven Architectures for Data Engineers
29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline