Hitting the Target(s) of Data Orchestration
summary
Builds a reproducible pipeline that pulls sensitive clinical and genomic views from a secure data lake, writes public summaries to AWS S3 persistent storage, loads precomputed summaries into a Redshift cluster, and runs daily from Posit Connect using one line to trigger target::make. Uses the R package Targets + Posit Connect to infer a directed acyclic graph, skip unchanged steps so only affected nodes are recomputed, and run parallel workers (crew). Outcome: secure, scalable orchestration for billions of rows that avoids full recomputation.
AI-written summary · about
chapters (12)
- 0:00Data orchestration challenges
- 1:54Data pipeline requirements
- 3:21Security in Posit Connect
- 6:06Recomputing pipelines problem
- 7:31DAGs for efficient workflows
- 9:20Job-based workflows limits
- 10:46Targets framework benefits
- 11:51Targets script structure
- 13:17Targets storage and computing
- 15:29Deployment in Posit Connect
- 15:51Refactoring pipelines
- 17:38Q&A
related talks
42mTransform your data approach: Develop a modern data strategy
21mPosit and the modern data stack: Using Posit, Snowflake, and Databricks
45mResilient Predictive Data Pipelines
38mBI DataOps Development Process
17mBuilding a data science pipeline for the FT with RStudio Connect
07mCloud architecture for population-scale analyses of genomes and AI.
30mData Engineering building your BI infrastructure from scratch
34mData pipelines powered by Open source and fun