DBTB INT Prakash Chockalingam r
summary
Practical checklist for running Spark Streaming pipelines on the Databricks platform with attention to production-scale concerns: how to handle state and backpressure, recommended microbatch sizes and checkpointing strategy, deployment patterns to move from development to production, and common failure modes observed with customers. Claims specific production results, tooling choices, and operational steps rather than high-level theory, with examples of pipelines that scaled and caught unexpected data-driven surprises.
AI-written summary · about
related talks
24mDistinguishing features of production quality pipelines
40mProductionizing Big Data - Stories from the Trenches
45mValidating Big Data Jobs - Stopping Failures before Production
41mSpeeding Up Data Pipeline Construction in Apache Spark
21mAvoiding Spark Pitfalls at Scale
42mBuilding Robust Streaming Data Pipelines with Apache Spark
57mScaling Databricks to Run Data & AI Workloads on Millions of VMs
40mTop 10 data engineering mistakes #bbuzz