From Gradients To Governance: Making PyTorch Lineage-Aware
summary
Argues that lineage must live inside PyTorch as a runtime constraint, not just post hoc logs, and shows an integration path using Open Lineage + Marquez with MLflow and Kubeflow emitting events from DataLoader/Dataset stages; proposes optional tensor metadata (origin, consent, jurisdiction, parent pointer) plus an internal policy store for enforcement at inference, notes autograd is the wrong place, cites a timeline to 2028 and 70% prioritize sovereignty, and flags distributed/mixed-data and attestation challenges.
AI-written summary · about
chapters (9)
- 0:00PyTorch lineage awareness
- 1:05Enterprise and regulatory deadlines
- 2:54Data sovereignty in PyTorch
- 4:57Failures from missing lineage
- 6:41Limitations of lineage tools
- 8:51Open Lineage for observability
- 12:48Integrating lineage tracking
- 18:51Technical approaches
- 20:40Open questions
related talks
32mTruly Open Lineage
27mData Lineage: A Key Building Block For A Successful Data Governance Strategy
30mCross-Platform Data Lineage with OpenLineage, A Foundational...
46mCross-Platform Data Lineage with OpenLineage
34mColumn-level lineage is coming to the rescue
16mArchitecting Scalable Data Lineage Graph for Privacy Compliance and Agentic Analysis
26mCompleting the machine learning loop
09mMLOps-Native Data Governance for Generative AI Pipelines