Data Mapping at a Billion Dollar Self-Driving Vehicle Startup
summary
Cruise describes an automated data-mapping system that scans BigQuery schemas and sampled content across ~14 PB, millions of tables and tens of millions of columns, surfacing thousands of sensitive-data findings and a surprising 55% of sensitive datasets abandoned by owners. A web UI called Indiana shows per-column labels; content detectors use Google Cloud DLP plus tuned post-processing; secure sampling runs through a Travel service; a human-in-the-loop CLI verifies labels and policy-tag soft-deletion locks down abandoned data.
AI-written summary · about
chapters (14)
- 0:00Data mapping at Cruz
- 2:15Privacy challenges
- 3:17Past data mapping failures
- 4:20Data map MVP design
- 5:21Indiana web application
- 7:50Labels and sensitive filters
- 8:56Sensitive data visualization
- 10:21Metadata pipeline scanners
- 11:25Scaling content scans
- 13:49Secure data sampling
- 14:53Human verification findings
- 15:13Abandoned data archiving
- 16:18Soft deletion with tags
- 17:01Privacy engineering impacts
related talks
46mProtecting sensitive data in huge datasets: Cloud tools you can use
40mProtecting sensitive data in huge datasets #bbuzz
10mDemystifying Access Control in the Modern Data Warehouse
28mBig Data: From Testing to Governance
52mBuilding the World's Largest Enterprise Data Warehouse with BigQuery
47mData Warehousing Migrations: Lessons from Home Depot
23m‘Samen Slimmer met Data’ – A company wide Data Transformation
42mDriving Data Democratization