Building Scalable and Extendable Data Pipeline
summary
Pipeline handles petabytes in a data lake and processes tens to hundreds of thousands of messages per second using Apache Kafka as the backbone; ingestion uses HTTP endpoints, optional proxying, batching and compression, plus dynamic sampling and per-user throttling to control cost. A protobuf message envelope with a schema ID in a custom schema registry enforces formats and enables deduplication via client-supplied message IDs. Multiple Kafka clusters separate ingestion, processing and serving; an S3 archiver plus work-queue fanout solves partition and reprocessing limits.
AI-written summary · about
chapters (43)
- 0:00Introduction
- 0:29Data Pipeline
- 1:16Scale
- 1:36Scalable and extendable pipelines
- 2:51Apache Kafka
- 3:02What is Kafka
- 3:47Scaling
- 4:31Producers
- 9:26Buffer
- 10:10Maintenance
- 10:41Kafka Proxy
- 10:51Kafka Behind the Scenes
- 11:10Basic Enrichment
- 11:29HighPerformance
- 12:29Partitioning
- 13:06Architecture
- 14:40Partitions
- 16:02Tuning
- 16:19Clusters
- 17:07Simple Chaining
- 19:06Data Pipeline Team
- 19:47Final Remarks
- 20:34Scaling the Pipeline
- 20:44Keeping Partitions and Topics Low
- 21:41Topic Naming Convention
- 25:20Proper Solution
- 25:32Data Type
- 25:50Data Type Comparison
- 26:25Metadata
- 26:46Consumer of Data
- 27:07Dynamic Fields
- 27:47Efficient Utilization
- 28:27Stream Processing
- 29:22Filtering
- 30:33Message Schema
- 30:44Message Formats
- 31:23Processing Data
- 31:50Message Envelope Convention
- 33:44Schema Registry
- 35:05Summary
- 36:29Slide
- 36:51Fun story
- 37:39Question
related talks
31mOne Data Pipeline to Rule Them All
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success
42mBuilding a Data Pipeline with Clojure and Kafka
38mData Pipeline with Apache Kafka
42mBuilding Robust Streaming Data Pipelines with Apache Spark
31mBridge to the Cloud
38mKafka Message Broker Under High Load
34mBuilding and Scaling Robust Zero-code IoT Streaming Data Pipelines