Automating Security in Your Data Pipeline
summary
Carta shows an actual pipeline: Apache Airflow runs dbt inside Docker to orchestrate ELT into S3 and Redshift, with Looker/Jupyter as consumers. Engineers build a Python blacklist of forbidden columns and run DBT tests for sensitive columns that alerted to three new offending fields. Access is provisioned via Terraform modules and a Gatekeeper API; schema changes use Sequel Migrate; KMS issues data keys and PG crypto/AES UDFs keep birthdates encrypted end-to-end.
AI-written summary · about
chapters (6)
- 0:00Introduction and welcome
- 3:33Why privacy matters now
- 17:35Ethics in privacy systems
- 21:46Building privacy tools
- 39:43Terraform and access control
- 58:41Summary and key takeaways
related talks
05mDump the Ducktape: Unblocking data pipelines at scale in healthcare using
49mdbt: A Software Engineering Approach To The Analytics Workflow
20mDataEngBytes 2023 PER-T1-10 - Matthew Carter
01h 28mA Guide to the Modern Data Stack
01h 42mFall in Love with Data Transformation: DBT as Your Weapon
21mSnowflake and dbt -- Our Journey to the Cloud
31mRings of Security for Big Data in Cloud
19mRobust CI Workflow with dbt for Data Warehouse Optimization