Tag! You're I.T.
summary
NIB scans and inventories over 300,000 column-level objects using Snowpark Python, then runs Snowflake SYSTEM$CLASSIFY (sampling up to 10,000 rows) to produce recommended semantic tags; a DBT-focused report maps recommendations to internal categories and flags misalignments. An auto PR generator uses the GitHub REST API to create or edit schema.yml files and open pull requests in batches (~100 models per run). The pipeline runs continuously, fixes many gaps and lowers exposure, while tagging work and reviews remain ongoing.
AI-written summary · about
chapters (7)
- 0:00Data tagging challenges
- 3:45Manual tagging at scale
- 8:03Inventory of classifications
- 11:14Snowflake classify recommendations
- 16:39Automating PR generation
- 21:14Securing downstream buy-in
- 23:46Recap and future steps
related talks
21mSnowflake and dbt -- Our Journey to the Cloud
28mScalable AI & Web Insights with GitLab, Snowplow & Snowpipe
29mBuilding End to End Gen AI Powered Analytics: From Data to Production
22mCase Study in Bridging Production Software and Data Practices for LLM Model Training
29mManaging your Data Assets: The interview
24mDataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi Farah
20mDataEngBytes 2023 PER-T1-10 - Matthew Carter
31mAdvancing security analytics with Snowflake and AWS at Snyk (PRT329)