Distinguishing features of production quality pipelines
summary
Practical checklist for taking Scala/Spark proofs of concept to production: prefer append-only data sources or versioned snapshots to avoid lost records; enforce reproducible deterministic outputs with high-watermark inputs and repeatable runs; add automated column and distinct-count validations and test on full datasets to catch regressions early; design idempotent writes or atomic renames for safe retries and aim for atomic deployments; only adopt lambda architecture when sub-minute freshness justifies the added complexity.
AI-written summary · about
chapters (9)
- 0:00Introduction to production pipelines
- 2:47Consistency in results over time
- 8:08Validating data results
- 10:35Reproducibility in development
- 13:47Caching and intermediate data
- 15:11Atomic deployments
- 17:04Idempotency in delivery
- 19:10Faster updates with Lambda
- 20:15Key takeaways and Q&A
related talks
45mData pipelines from zero to solid
03mDBTB INT Prakash Chockalingam r
21mAvoiding Spark Pitfalls at Scale
45mValidating Big Data Jobs - Stopping Failures before Production
45mResilient Predictive Data Pipelines
31mOne Data Pipeline to Rule Them All
18mData quality: the key to long term happiness
30mDesigning Resilient Data Pipelines