AI-driven DataOps: Building data products with a taste of the future
summary
GitLab Data Team experiments four architectures: CSV cloud API prototyping, cloud connectors to Snowflake via MCP, Snowflake-first paths using Snowflake Cortex Search and Snowflake Cortex Complete/text-to-SQL, and a POC combining code, a semantic layer, and a vector DB (VV8) running in Snowflake containers. Key claims: semantic metadata fixes many LLM errors, finance queries work reliably on ~10 curated tables but fail to scale, about 80% of AI projects fail, there are 107 queued AI use cases, and security plus unknown AI TCO remain blocking issues.
AI-written summary · about
chapters (22)
- 0:00AI-driven DataOps intro
- 1:03Why AI matters in data
- 2:29AI hype cycle explained
- 5:03DevSecOps role in data ops
- 6:27DevSecOps platforms evolution
- 8:14Data, DevSecOps, AI trinity
- 9:16DataOps challenges and metrics
- 10:20Personal AI and data journey
- 11:03First iteration: Cloud API CSV
- 12:05Second iteration: MCP Snowflake
- 13:31Missing semantic layer in AI
- 14:14Third iteration: Snowflake Cortex AI
- 16:19Fourth iteration: Cortex Analyst SQL
- 17:21Fifth iteration: Cortex Complete AI
- 18:44Sixth iteration: Vector LLM POC
- 21:30Key lessons and AI strategy
- 22:36Quick wins vs long-term AI
- 23:39Future of AI data products
- 24:21Q&A: Semantic layers & vectors
- 27:36Q&A: Embedding data & semantics
- 29:21Q&A: Pipeline design Snowflake vectors
- 30:47Closing remarks
related talks
30mBuilding Data AI Application with the Snowflake AI Data Cloud
28mScalable AI & Web Insights with GitLab, Snowplow & Snowpipe
29mBuilding End to End Gen AI Powered Analytics: From Data to Production
28mUsing Dataops To Build And Manage Data Products On Snowflake
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
27mAgile Data Platforms : Innovate at the speed of thought
15mEnterprise AI Runs on Trusted Data
27mDataOps & Full Snowflake Automation In 1 Click