Interactively Analyse 100GB of Data
summary
Live demo that interactively analyses a 100GB Open Library dump using Apache Spark on Amazon EMR with an Apache Zeppelin notebook, running the same map/filter/reduce Python code locally and on a cluster. It shows S3 ingest, Parquet and caching tradeoffs, cluster provisioning and monitoring (example clusters up to hundreds of nodes), practical performance tips and that exploratory queries, histograms and aggregations complete interactively on the 100GB set.
AI-written summary · about
chapters (31)
- 0:00Big data challenges
- 1:05Big data systems history
- 2:28River and agenda
- 3:11Three Vs of big data
- 5:19Twitter data flood
- 6:23Data structure variety
- 8:34Big data processing problems
- 10:18Latency and consistency
- 11:25Batch vs real-time alerts
- 12:31Scaling up vs out
- 13:12Single vs distributed systems
- 15:22Flexibility vs fault tolerance
- 16:48Cluster management
- 17:50Apache Spark ecosystem
- 18:33MapReduce vs Spark
- 20:00Spark API basics demo
- 21:25Reading large JSON in Spark
- 24:12RDD transformations
- 26:10Filtering and counting entries
- 27:37Aggregating with reduce/map
- 30:10Visualizing results
- 32:13Interactive data exploration
- 34:22Scaling up with Amazon Elastic MapReduce
- 35:46EMR cluster setup
- 37:11EMR job monitoring
- 39:21Job results and metrics
- 40:55Spark job optimization
- 41:39Cluster configuration
- 42:47Parquet for efficiency
- 43:29Approximate query systems
- 44:10Closing and resources
related talks
16mLogging Apache Spark - How we made it easy
42mApache Zeppelin
13mLogging Apache Spark How we made it easy
35mScaling up to Big Data Devops for Data Science
39mBig Data Systems Performance: The Little Shop of Horrors
34mKick-Ass Tools and Processes for Scalable Data Science
17mCarlos Guestrin, CEO of Dato Inc. @ MLconf SEA
47mHadoop: Lessons Learned