Supercharge Your Real-Time Analytics: Snowpipe Streaming & Dynamic Tables in Snowflake
summary
Live demo builds a real-time pipeline that injects row-level events directly into Snowflake using Snowpipe Streaming and transforms/aggregates them with Dynamic Tables. A Java CDC simulator pushes data (no external Kafka required for the demo), showing 1 million rows in 8 seconds into a variant-based staging table; dynamic tables auto-refresh with configurable target_lag and incremental refreshes, preserve event order, and surface metrics for monitoring. The setup works end-to-end and highlights costs and clustering tradeoffs.
AI-written summary · about
chapters (9)
- 0:00Real-time analytics intro
- 4:44Snowflake architecture
- 9:20Traditional vs Streaming
- 22:46Snowpipe Streaming demo
- 44:40Dynamic Tables intro
- 58:14Creating Dynamic Tables
- 1:07:07Best practices
- 1:11:21Real-time use cases
- 1:13:48Q&A
related talks
01h 30mSnowpipe Fundamentals: Ingesting Streaming Data into Snowflake Made Easy
28mBuilding A Governed Data Lake
24mBreak Barriers In Real-time With Data Streaming
23mBuilding A Code Generated Data Platform
14mStreaming Data Pipelines to Enhance Real-Time Decision-Making
32mScalable Python and SQL Data Engineering without Migraines
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
13mLightning Talk: Highlights of the Snowflake Elastic Data Warehouse