Scaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...

on YouTube ↗31m

" summary

Cloud-agnostic Juicedata file system runs on Kubernetes, mounts object storage as a POSIX volume and handles 100 PB / 100 billion files by pushing capacity to S3 while scaling metadata horizontally with pluggable engines (Taiki recommended). Clients use FUSE with multi-level caching, a CSI driver for PVCs and support POSIX + S3 + HDFS APIs across thousands of mixed GPU/CPU nodes. Real deployments report ~50 GB/s average throughput, 12k clients, and saved ~60% storage cost versus prior commercial systems.

AI-written summary · about

" chapters (10)

1. 0:00Introduction
2. 1:52Data explosion challenges
3. 9:45Storage dilemma
4. 13:27JuiceFS architecture
5. 18:50Optimizing IO patterns
6. 23:30Multicloud support
7. 25:18Case study data silos
8. 26:52Case study 100 PB scale
9. 28:21Recap and limitations
10. 29:44Open-source and Q&A

" related talks

• 28mData @Scale Seattle- Brian BeachBrian Beach · Data @Scale Seattle 2015
• 29mCubeFS in Action: Empowering Users Through Case StudiesLeon Chang · KubeCon + CloudNativeCon Europe 2025
• 33mSponsored Session: Open Source Cloud Native Storage Solutions for Stateful Applications from ...Rahul Sharma · ContainerCon @ Open Source Summit North America 2022
• 29mAccelerate your cloud architectures with the right storage optionsJune Park · AWS Summit ANZ 2023
• 33mThe Evolution and Innovation of CubeFSLeon Chang · KubeCon + CloudNativeCon North America 2022
• 14mFalconFS: Distributed File System for Large-Scale Deep Learning PipelineXinwei Hu, Lu Zhang, Haibo Chen, Mingkai Dong, Jingwei Xu, Junbin Kang, Mingyu Liu, Shaohong Guo, Ziyan Qiu, Mingzhen You, Ziyi Tian, Anqi Yu, Tianhong Ding · NSDI '26
• 45mOpen Storage in the Enterprise with GlusterFSDustin Black · LinuxCon + CloudOpen Europe 2013
• 24mCloud Workloads At Petabyte ScaleSrini Srinivasan, Amit Shah, Merritte Stidston · Aerospike Digital Summit 2021