The State of Data Engineering
summary
Claim: about 85% fail 15% succeed on data projects (Gartner) and measured team value trends are sliding down not up. Evidence: surveys show 80% of data scientists on engineering tasks and generative AI usage splits — 55% use gen for code gen 46% for debugging 12% for data processing 25% not using it. Tools and tradeoffs named: Airflow, dbt, Kafka/Pulsar (pub‑sub vs point‑to‑point), managed Kafka; vendors simplify ops not architecture. Concrete advice: pay for real engineers, measure business value, mentor and talk to the business.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 7:32Data warehousing history
- 14:58Data science team mistake
- 26:14Data engineering confusion
- 33:40Generative AI in data engineering
- 40:25Solving inefficiency and value
- 44:01Unsolved problems and alignment
- 49:05Call to action
related talks
32mWhy Most Data Projects Fail and How to Avoid It
25mWhy Software Engineering Best Practices Fail in Data Engineering
31mWhy Most Data Projects Fail and How to Avoid It
39mFoundations of Data Teams
41mData engineering without walls
41mData engineering in 10 years
37mEngineering data quality
44mThe Art of Data Engineering