Your data fits in RAM: How to avoid cluster computing
summary
Claim: most “big data” workloads run on one machine if enough RAM is rented in the cloud. Evidence: AWS offers up to 768 GB instances and rentable GPUs for ~\$0.90/hr (often \$0.30/hr). Tools and results: columnar binary formats like Parquet fivefold reduction (11 MB TSV → 2.7 MB), sparse matrices cutting 96 MB dense to 186 KB, and multiprocessing turning a 12‑hour job into 12h to 20m, all used and shown to work.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 2:12Talk inspiration
- 5:45Key takeaways
- 6:48Why the cloud is your friend
- 8:37How to use the cloud effectively
- 9:44Use familiar tools
- 11:30Do you really need all your data?
- 15:25File formats matter
- 19:21Sparse matrix optimization
- 22:50Your laptop is stronger than you think
- 24:16CPU core leverage
- 26:02Optimize everything
- 27:26Summary
- 28:10Q&A
related talks
25mGreenOps at Scale: The 50TB Memory Diet for a Petabyte Platform
30mBlazing fast I/O of data in the cloud with Daft Dataframes
51mIn-Memory Computing - The Big Picture
29mHow to Build a Cluster to Run Hundreds of Thousands of High-CPU/High-MEM Jobs
38mCharacterizing the performance of Big Data processing platforms
39mDensity! The not-so-secret trend driving the future of Data Analytics
01h 06mBest practices when accessing Big Data or any other data!
45mInteractively Analyse 100GB of Data