Data Engineering in the AI Era: Building a Robust Foundation for AI Applications
summary
Walks through a production stack that uses Debezium CDC into Kafka, Snowflake as the warehouse and Snowplow for event tracking, with DBT models, Kafka single‑message transforms for PII masking, and dev containers for reproducible ops. Demonstrates incremental snapshots via signaling, dead‑letter topics for bad events, and a Snowflake migration that cuts compute/storage costs ~30% versus Redshift and reduces vendor spend by $20K annually while keeping full data ownership.
AI-written summary · about
chapters (13)
- 0:00Data engineering in AI era
- 3:30Business deliverables & AI
- 8:54Solid data foundation for AI
- 9:16Data ingestion strategies
- 12:52Change data capture with Debezium
- 26:26Sync connectors & Snowflake
- 35:37Event tracking with Snowplow
- 46:51Choosing data platform
- 52:55Data modeling with DBT
- 57:12Handling sensitive data
- 1:00:08Data ops & Dev containers
- 1:03:49Pipeline signaling & automation
- 1:13:16Summary
related talks
30mBuilding Data AI Application with the Snowflake AI Data Cloud
30mData Engineering building your BI infrastructure from scratch
34mTales of Data Architecture Evolution
35mHigh-performance data foundation for AI success
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
31mAI-driven DataOps: Building data products with a taste of the future
18mBehavioral Data Creation for AI - Delivering Value and Insights
29mBuilding End to End Gen AI Powered Analytics: From Data to Production