Pachyderm: Unlock the Power of Kubernetes for Big Data
summary
Pachyderm runs an open-source distributed processing platform entirely on Kubernetes, using the Kubernetes Jobs API to schedule thousands of parallel containers and treat object storage like a CDN for shards. It implements version-controlled data (commits, diffs, branches) backed by a metadata DB on persistent volumes so teams get reproducibility, instant revert and data lineage for petabyte-scale datasets. Demos stream logs into commits, spawn parallel jobs, and recover failed shards automatically.
AI-written summary · about
chapters (4)
related talks
19mModern Data Science in a Cloud Native World
08mProcessing Robot Data at Scale with R and Kubernetes
33mCompliant Data Management and Machine Learning on Kubernetes
28mProgress For Big Data In Kubernetes
47mData Mesh in Kubernetes
22mExploring Cloud Native Big Data Platform in SPDB
39mEnd to End Machine learning pipelines for Python driven organizations
35mModern Big Data Pipelines over Kubernetes [I]