Big Data in Java World: Apache Spark and Scala. Arch and live coding

on YouTube ↗01h 12m

" summary

Overview of hands‑on Apache Spark with Scala: covers architecture, client vs cluster driver/executor modes and live coding in Jupyter (Almond) using RDD / DataFrame / data set (typed) vs data frame APIs. Demonstrates reading and writing parquet columnar files, collapsing partitions to a single file, running a word‑count and joins, and statistical queries on a ~1M+ rows US flights dataset (2018) to compute correlation, group counts and averages — results run successfully and return Delta as the largest carrier.

AI-written summary · about

" chapters (46)

1. 0:00Intro
2. 4:21Overview
3. 6:00What is Big Data
4. 12:35What is Apache Spark
5. 14:17Spark vs Hadoop
6. 16:10Architecture
7. 17:37Spark Session
8. 18:43Spark Submit
9. 19:33Spark Deploy Mode
10. 21:04Spark Data Structures
11. 22:06Spark Data Structure
12. 25:15Spark Data Frame
13. 25:43Spark Data Set
14. 27:27DataFrame Functions
15. 30:30Transformations
16. 31:39Actions
17. 32:04pandas vs Spark
18. 33:36Spark with text files
19. 34:11Row oriented vs column oriented
20. 35:36Clickhouse tool
21. 36:10Types of data formats
22. 37:31Pocket
23. 38:30Recap
24. 38:52Formats
25. 39:53Spark
26. 40:26Scala
27. 41:50Spark Context
28. 42:15RDG
29. 42:50RDG map
30. 43:37RDG result
31. 44:00RDG conversion
32. 44:46Data Structure
33. 46:13Single Partition
34. 46:44File Function
35. 47:43DataFrame
36. 50:11Real life example
37. 50:56Review
38. 51:38Data scientists
39. 52:05DataFrame to DataSet
40. 52:30Typed Storage
41. 52:53Imports
42. 53:31Airport
43. 53:46Correlation
44. 54:49Extract Data
45. 55:11Define Statistical Features
46. 55:30Simple Statistical Features

" related talks

• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler · GOTO Chicago 2016
• 46mScala and the JVM as a Big Data PlatformDean Wampler · Scala World 2015
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler, Andy Petrella · Scala Days Berlin 2016
• 45mScala and the JVM as a Big Data Platform Lessons from Apache SparkDean Wampler · Scala Days Berlin 2016
• 52mSolving Analytical Problems using Apache SparkRohit Bhardwaj · Developer Summit 2020
• 50mIntroduction to Real-Time Big Data with Apache SparkTaras Matyashovsky · JavaDay Lviv 2015
• 01h 36mSpark for Scala DevelopersBrian M Clapper · Scala World 2017
• 59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & AkkaHelena Edelson · Philly ETE 2015