Dump the Ducktape: Unblocking data pipelines at scale in healthcare using
summary
Describes a production data-protection pipeline that implements dynamic data masking across 10,000 tables and 100,000 columns, wiring DBT artifacts and Snowflake metadata to generate masking SQL; a DBT-only run takes ~2 hours but pushing heavy work into DuckDB drops execution to about 5 minutes (most time remains Snowflake I/O). Also shows DuckDB as a pipeline engine: SQLAlchemy → Arrow → DuckDB → Parquet on S3 → Snowflake, with Datadog tracing and ongoing DuckDB profiling.
AI-written summary · about
related talks
06mA quack at building scalable data pipelines with DuckDB
26mBuilding Data Lake Using DuckDB
01h 00mAutomating Security in Your Data Pipeline
21mSnowflake and dbt -- Our Journey to the Cloud
41mAccelerating privacy-enhancing data processing
28mAnalyze data easily with duckdb - and the implications on data architectures
01h 42mFall in Love with Data Transformation: DBT as Your Weapon
27mDataOps & Full Snowflake Automation In 1 Click