Accelerating Spark Workloads in a Mesos Environment with Alluxio
summary
Alluxio deployed on Mesos (via DC/OS) sits between Spark and backing stores (S3, HDFS), providing a memory-speed virtual distributed storage cache and a unified namespace. Demo mounts an S3 bucket and a 1 GB sample file, first Spark read takes ~30 s, a second read takes ~3.5 s (≈8× faster) because data is cached in Alluxio; a production user reports some queries sped up ~300×. Installation uses the DC/OS Alluxio package and standard HDFS/S3 connectors.
AI-written summary · about
chapters (16)
- 0:00Start
- 1:20Data Ecosystem Yesterday
- 1:46Data Ecosystem Today
- 2:31Data Ecosystem Issues
- 3:00Data Ecosystem with Alluxio
- 4:30Next Gen Analytics with Alluxio
- 5:25Enabling Next Gen Analytics
- 6:08Fastest Growing Big Data Open Source Project
- 9:32+ Sharing Data via Memory
- 10:54+ Data Resilience During Crash
- 11:58Alluxio Architecture
- 12:47Alluxio Client
- 13:58Alluxio Master Master is responsible for managing med
- 15:27Alluxio on DCIOS
- 17:30Demo Setup
- 27:48+ Conclusion
related talks
31mAccelerating Spark Workloads in an Apache Mesos Environment with Alluxio
40mOpen Source Memory Speed Virtual Distributed Storage System
45mAlluxio Data Orchestration for Machine Learning
32mStreamlining Data Access for with
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed...
39mThe Sparkling Solution that Scales to IoT
52mSolving Analytical Problems using Apache Spark
42mApache Spark on planet scale