How DTCC achieves data resiliency with Snowflake and AWS (PRT272)
summary
DTCC describes a production business-continuity system built on Snowflake and AWS that replicates 42 petabytes replicated across regions, serves ~700,000 queries/day, and runs ~15 disaster‑recovery exercises/year. It uses Snowflake database and account replication, private‑link dual‑written S3 stages, and a client redirect global URL requiring a CNAME change so failover completes in 30–45s DNS switch. Pipelines are replayable to avoid duplication, automation reduces onboarding to 2–3 days, and operational P99 replication lag is under two minutes.
AI-written summary · about
chapters (16)
- 0:00Intro
- 3:17Consequences of real-world disaster events
- 6:09What do we mean by business continuity?
- 8:27Snowflake architecture and AWS
- 9:24Preparing for different types of failures
- 9:54Snowflake enables business continuity
- 16:02Disaster recovery flow
- 17:35Account replication and failover groups
- 21:40Seamless identity and user authentication
- 27:45App Rotation Model in Snowflake (sample use case)
- 29:00Flexibility for different failover scenarios
- 29:36Connection global URL for application rotation/client redirect
- 30:11Replay missed data loads after DR
- 31:03App rotation for extended runs - Process flow
- 34:24Replication orchestration
- 36:19Benefits of Snowflake replication and failover for DTCC
related talks
23mMaturing The Snowflake Data Cloud At London Stock Exchange
20mSnowflake concepts & architecture
06mLightning Talk: Performance Testing of FoundationDB using TPC-C
13mLightning Talk: Highlights of the Snowflake Elastic Data Warehouse
30mA Devops Approach to Data: A Tale Through the Trenches in Search of a Cloud Data Revolution
57mDSS Keynote — Back to the Future: Prepare Your Data Infrastructure for Anything
28mBuilding A Governed Data Lake
37mEnabling Resilience through the Cloud: AWS's Disaster Recovery Solution