Speeding Up Data Pipeline Construction in Apache Spark
summary
How to speed up building large-scale data pipelines with Apache Spark by closing the gap between interactive experimentation (Spark Shell/Notebooks with sample datasets) and production jobs that run scheduled at full scale. Covers architectural patterns for secure data access, scalable pipeline execution, using a plugin-based system with Spark SQL to boost developer productivity, designing data lakes for pipelines, and recording pipeline execution history for analysis and anomaly detection.
AI-written summary · about
related talks
42mBuilding Robust Streaming Data Pipelines with Apache Spark
52mSolving Analytical Problems using Apache Spark
21mAvoiding Spark Pitfalls at Scale
42mApache Spark Declarative Pipelines in Action
36mNo more struggles with Apache Spark (PySpark) workloads in production
24mTaming Data Skew in Production ML Pipelines
01h 35mDeep Dive into the Big Data Landscape
45mFrom Big Data to Fast Data: Apache Spark