Accelerating Spark Workloads in an Apache Mesos Environment with Alluxio
summary
Alluxio on a Mesos / DC/OS cluster frontends HDFS and S3 into a unified namespace, caches blocks in memory (demo uses Alluxio 1.5, Spark 2.x on EC2) and keeps data after a Spark context dies. A 1 GB file (500 MB blocks, two blocks) measures 31 s cold and 3.6 s hot—an 8x repeated-access speedup. Use cases report up to 300x peaks and the system decouples data lifetime from applications.
AI-written summary · about
chapters (17)
- 0:00Start
- 0:35+ Outline
- 0:58+Data Ecosystem Yesterday
- 1:34Data Ecosystem Today
- 2:05Data Ecosystem Issues
- 2:45Data Ecosystem with Alluxio
- 3:31Next Gen Analytics with Alluxio
- 3:48+Enabling Next Gen Analytics
- 4:17Fastest Growing Big Data Open Source Project
- 6:02+Machine Learning Case Study
- 7:31+ Consolidating Memory
- 9:49+Data Resilience During Crash
- 12:21Alluxio on DC/OS
- 14:00+ Demo Environment
- 14:29+ Demo Setup
- 26:26+ Results
- 27:28+ Conclusion
related talks
35mAccelerating Spark Workloads in a Mesos Environment with Alluxio
40mOpen Source Memory Speed Virtual Distributed Storage System
45mAlluxio Data Orchestration for Machine Learning
32mStreamlining Data Access for with
39mThe Sparkling Solution that Scales to IoT
05m⚡ Lightning Talk: Using Decentralized File System to Optimize ML/AI Workloads on Kubernetes
42mApache Spark on planet scale
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed...