Current Methods for Building Data Pipelines
summary
Panelists compare concrete stacks and tradeoffs: lots of orgs still ingest via SFTP into object stores, orchestrate with Airflow + Spark + DBT on object stores, some moved heavy transforms from Spark to DBT and report measurable quality and performance gains, others run Apache Flink for streaming and argue for Kafka as a single-source data lake to reduce copies, many warn schema changes and metadata gaps break pipelines, and one claimed DBT testing cut regressions after migration.
AI-written summary · about
chapters (12)
- 0:00Data pipeline methods intro
- 6:10Tools and stacks today
- 11:44Batch vs streaming trends
- 19:47Real-world pipeline examples
- 26:17Future pipeline challenges
- 35:47What pipelines need
- 40:45Q&A on ingestion
- 45:45Virtual ontologies
- 50:22Data modeling approaches
- 53:13Security in pipelines
- 56:50Streaming necessity
- 1:00:00Closing remarks
related talks
19mDemystifying Modern Data Pipeline Architecture
41mModern Data Architectures – What They Are and How to Implement?
47mHow to pick the best Data transformation strategy
26mDataOps: Accelerating Digital Transformation
59mPanel: Future of Data Services
58mTales of Data Architecture Evolution
21mDesigning Modular Data Pipeline
42mBuilding a Streaming (E-health) Data Pipeline: When and How?