Scalable AI & Web Insights with GitLab, Snowplow & Snowpipe
summary
GitLab builds a CI/CD-managed pipeline using Terraform, Snowplow (collector & enricher), S3 staging, Snowpipe for continuous loads into Snowflake, DBT transforms and Tableau dashboards. The system handles 110 million events/day, lands about 400 GB raw data/day and stores ~30 TB in Snowflake. Snowplow validates and enriches events, Snowpipe auto-loads files via SQS notifications, separate compute avoids noisy neighbors, and the stack autos-scales to control cost while delivering near‑real‑time web and AI-productivity metrics.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:51GitLab mission and data team
- 3:56AI and analytics challenges
- 6:23Modern data stack overview
- 8:34Snowplow data enrichment
- 11:25GitLab and Terraform
- 13:13Snowflake data warehouse
- 14:36Snowpipe real-time ingestion
- 16:03End-to-end pipeline architecture
- 21:44Integrated approach benefits
- 24:55Pipeline performance and conclusion
related talks
40mDo the Magic with All-Remote Data Teams...
31mAI-driven DataOps: Building data products with a taste of the future
25mHow we create and leverage Data Services in GitLab.com
29mBuilding End to End Gen AI Powered Analytics: From Data to Production
21mSnowflake and dbt -- Our Journey to the Cloud
27mDataOps & Full Snowflake Automation In 1 Click
30mBuilding Data AI Application with the Snowflake AI Data Cloud
28mUsing Dataops To Build And Manage Data Products On Snowflake