Jason Huang, Solutions Engineer, Qubole
summary
Company history notes Hive co‑founders and frames cloud as separating storage from compute, claiming elasticity cuts idle clusters; cites industry numbers: 13% reach full production, 57% report a skills gap. Demo runs k-means on Spark MLlib and a logistic example using SparkR and PySpark, reading data from S3 storage + EC2 compute, returning clustering results and query history via Qubole’s managed clusters so infrastructure setup is avoided and jobs succeed.
AI-written summary · about
chapters (8)
- 0:00Start
- 0:21Work at Facebook inspired the founding of Qubole
- 2:55State of the Big Data Industry (-417)
- 4:10Apache Spark
- 5:23Common Spark Use Cases
- 8:01How about R? Use SparkR
- 8:54What about the cloud?
- 12:49DIY Getting Started on the Cloud
related talks
34mHive on Spark #bbuzz
49mPresto in the Cloud
33mKubernetizing Big Data and ML Workloads at Uber
28mFlexible Cloud, Whatever The Workload
05mHadoop in the cloud: A Nielsen use case
49mBuilding a 100‑Petabyte Analytics Warehouse
04mStartup Showcase: JethroData
02h 05mBig Data and the Data Lake from the Analyst Perspective