End-2-End Monitoring and Troubleshooting a Real-Time Data Pipeline
summary
A monitoring platform that centralizes Kafka, Spark, Storm, Flink, Cassandra and Elasticsearch metrics claims to make pipeline debugging measurable by modeling the pipeline as a first‑class object and organizing metrics by throughput, latency, error rate. It shows automated topology discovery and anomaly detection with health rollups, and demonstrates a real incident where an injected ingest server randomly drops documents causing Q size growth and downstream throughput collapse, which the pipeline view exposes.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:35What is a datafirst application
- 1:12The data pipeline
- 3:00Automated topology discovery
- 3:57Characteristics of data pipelines
- 5:22Data pipelines are interdependent
- 6:08Troubleshooting and debugging
- 6:22Put all your data in one place
- 7:02Organize concerns horizontally
- 9:25Hierarchy of concerns
- 10:50Example
- 15:17Summary
related talks
12mA Principled Approach to Monitoring Streaming Data Infrastructure at Scale
42mUnified Monitoring of Containers and Microservices [I]
19mObservability for Data Pipelines: Monitoring, Alerting & Tracing Lineage
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
35mApache Kafka as a Monitoring Data Pipeline
21mSweet Streams are Made of These: Applying Metrics-driven Development to Apache Flink
33mBeyond Monitoring: The Rise of Observability Platform
30mRunning Highly Scalable Tracing, Logging, Monitoring & Metric System in Production