The Evolution of Hadoop at Spotify Through Failures & Pain
summary
Spotify runs ~1,700 Hadoop nodes with 62 PB storage, ingesting ~30 TB logs/day and running ~20,000 jobs/day; an initial Python LogArchiver using SCP fails frequently so delivery moves to Kafka and latency drops from hours to seconds. Job orchestration standardizes with Luigi, a pure‑Python HDFS client (Snakebite) cuts scheduler latency, and JVM pipelines using Crunch run ~8x faster than Hadoop Streaming. NameNode HA and open‑file failover bugs cause multi‑day outages, so a pre‑production cluster, snapshots, S3 backups, monitoring and Zeppelin visualizations are added.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:22Overview
- 2:40What is Spotify?
- 4:09Powered by Data
- 7:23Moving Data to Hadoop
- 9:27LogArchiver
- 13:07Workflow Management Fail!
- 17:37Hadoop Availability
- 18:33How did we do?
- 20:32What happened in the last quarter?
- 24:47Lessons Learned
- 28:53Going from Python to JVM
- 30:01Moving from Python to Crunch
- 31:07Crunch vs Hadoo Streaming Benchmark
- 31:56Let's Review
- 33:00Growth of Hadoop vs. Spotify Users
- 33:28Explosive Growth
- 36:39Inviso
- 37:18Hadoop Report Card
- 38:45Apache Spark with Zeppelin
- 40:10Takeaways
related talks
57mBig Data Pipelines With Scala - Part 2 of 2
49mData Platform Architecture, Evolution, and Philosophy
13mLogging Apache Spark How we made it easy
45mSolving Real & Big (Data) Problems Using Hadoop
40mTaming Spiky Log Volumes: Maintaining Real-Time Log Accessibility with Kaldb
29mData Acquisition: Overcoming Challenges
34mChasing the Stream Processing Utopia
45mFrom Big Data to Big Information