How Data & Software Eng. Teams Collaborate to Ensure Smooth Data Integrations
summary
Claim: integrating a new product’s data into analytics is not plug‑and‑play. The Vault launch forces work on Stitch and Hevo extra SSH tunnels and read replicas, BigQuery + dbt transforms, and Lightdash dashboards; product requests 30–40 metrics that get pared to essentials. Surprises include missing fields, timezone and currency mismatches, hard deletes/archiving and legacy migration gaps, and test data leaking into prod. The fix is upstream enforcement—database constraints, Great Expectations or CI tests—and agreed CI/CD data contracts with SLAs for break fixes.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:46TL;DR
- 1:55About me
- 3:31Agenda
- 4:16The problem
- 6:421. Logistics
- 13:442. Infrastructure
- 19:293. Data model
- 26:404. Application & data flow
- 33:27Data contracts
- 38:13Wrapping up
- 41:48Outro
related talks
29mManaging your Data Assets: The interview
31mData Mesh & Data Vault
13mData for product
39mBREAKING YOUR DATA TEAM OUT OF THE SERVICE TRAP
48mBuilding a data mesh
26mApplying DevOps practices in Data and ML Engineering
31mHow to Implement DataOps using
33mAccidental DataOps: How we matured our organization with data, and how you can too!