Analyzing Pwned Passwords with Apache Spark

on YouTube ↗42m

" summary

Loads SHA-1 hashed breach records into Apache Spark via a Zeppelin notebook, infers schema from CSV/JSON, and joins with a 700k common-password list to produce counts and visualizations. Analysis shows obvious winners like "123456" seen >20 million times, length peaks at 6- and 8-character peaks, and a dog-name hit with Charlie: 278,000 matches. Uses DataFrame/Dataset APIs, UDFs and SQL DSL for filtering, and demonstrates practical ETL patterns and security takeaways (use password managers, prefer app-based 2FA).

AI-written summary · about

" chapters (15)

1. 0:00Start
2. 2:40Introducing Apache Spark
3. 5:34Abstractions
4. 6:18The Rdd
5. 8:25The Data Frames Api
6. 14:01Credential Stuffing
7. 17:07Csv Reader
8. 21:54Where Clause
9. 22:56Running Raw Sequel
10. 24:20Where Clauses
11. 25:35Contents of the Passwords
12. 28:40User-Defined Functions
13. 30:19Benefits of Spar
14. 34:01Security Implications of Working with Big Data
15. 41:19Clustering in Apache Spark

" related talks

• 40mAnalyzing Pwned Passwords with Apache SparkKelley Robinson · GOTO Chicago 2018
• 39mAnalyzing Pwned Passwords with Apache SparkKelley Robinson · Scala Days New York 2018
• 39mLive-Hack: Analyzing 7 years of Buzzwords at Scale #bbuzzChristoph Tavan · Berlin Buzzwords 2016
• 27mProfiling "VIP Accounts" Access Patterns in User-Centric Data StreamsRod Soto, Joseph Zadeh, Xiodan Li · BSidesSF 2019
• 01h 36mSpark for Scala DevelopersBrian M Clapper · Scala World 2017
• 35mDataFrames in Apache Spark - the analysts perspectiveMarcin Szymaniuk · Berlin Buzzwords 2018
• 19mApache Beam pipelines at 100TB+ scale using Apache SparkDavid Moravek · Berlin Buzzwords 2019
• 43mApache Zeppelin, the missing GUI for your Big Data back-endDuyHai Doan · Voxxed Days Bucharest 2016