The future of Data Engineering panel
summary
Panel argues the future mixes mastering fundamentals with AI augmentation, not magic fixes; claims 80–90% burnout among data engineers (used as a caution), shows concrete tech bets — Ibis and Iceberg interoperability and Spark handling multi‑petabyte jobs — and gives real examples: a REST API and pipelines built for a large service that end up handed to cybersecurity, mining sites forcing batch over streaming, and unresolved governance needs for data mesh and metadata (no single technology yet solves federation).
AI-written summary · about
chapters (30)
- 0:00Future of data engineering
- 1:10Fundamentals vs buzzwords
- 2:12Data engineering roles evolution
- 3:15Local challenges in data engineering
- 4:40Data skills and education trends
- 5:43Volume velocity variety
- 6:47Aligning data with business
- 8:12Data engineering as engineering
- 9:57Burnout in data teams
- 11:40Speed vs quality balance
- 13:07Defining data engineering
- 14:55Data vs software engineering
- 17:45Future of data roles
- 18:47Data mesh ownership
- 20:50Emerging niches
- 22:33Domain-driven design
- 25:02Data governance decentralized
- 27:50Quality vs decentralization
- 30:57Upskilling business users
- 33:50Undocumented systems risks
- 35:36Interoperability in tech
- 37:43Data ownership portability
- 38:46Consumer data rights
- 40:47Critique of data stack
- 42:10Tooling consolidation
- 44:56Data tools landscape
- 46:41Collaborative platforms
- 47:22Data fabric vs mesh
- 48:26AI replacing engineers
- 49:50Closing remarks
related talks
31mFuture of data engineering: what's in store?
51mPanel Debate: Rise Of The Data Engineer
28mDataEngBytes Panel discussion - Sydney
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
40mData Engineering in a Rapidly Changing World
59mPanel: Future of Data Services
38mThe Rise & Downfall of the Data Engineer REVISITED
25m"Data Engineering: The World Between Worlds"