How to build blazingly fast distributed computing like Apache Spark In-house?
summary
ClustrData examined Apache Spark and similar big-data engines and found they meet performance needs but not extreme cost constraints. The talk describes designing an in-house distributed processing framework that delivers comparable analytics with much lower resource usage by rethinking execution and resource trade-offs to run smaller clusters for shorter durations. It shares the implementation choices and lessons learned from building that cost- and efficiency-focused system.
AI-written summary · about
chapters (4)
- 0:00Start
- 7:36Internals of ReduceByKey Operation
- 9:33ReduceByKey (Post)Operation (In-Memory + Disk)
- 18:06Upendra Singh
related talks
26mBuilding Apache Spark Clusters on K8s for AI Workloads
26mOptimal Approaches for Real-Time Machine Learning with Apache Spark on Kubernetes.
26mRe-architecting the Data Platform with Spark on Kubernetes
36mTower of Babel: Making Apache Spark, Kubeflow, and Kubernetes Play Nice
22mSPOK - Managing ML/Big Data Spark Workloads at scale on Kubernetes
29mScaling Apache Spark on Kube to Apple Scale
24mDynamic Big Data in the Cloud with Kubernetes
39mSpark on Kubernetes: Best Practice and Performance