Hadoop in the Cloud: The Story of Migrating Hundreds of Petabytes
summary
Migration of OK’s massive Hadoop platform (over 200 PB in HDFS, ~70k vcores and ~200 TB RAM) into the cloud is described, covering both analytics and recommender-production workloads. The talk explains the migration challenges at petabyte scale and the technical approach used to move and operate Hadoop-based infrastructure in a cloud environment.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 2:13What is Hadoop
- 5:06Current infrastructure issues
- 7:15Cloud migration approaches
- 9:03Hadoop cloud architecture
- 13:37Building cloud cluster
- 16:24Data migration to cloud
- 17:07Data replication problems
- 20:20Faster data migration
- 21:01Migrating cluster masters
- 22:59NameNode issues
- 25:49Post-migration performance drop
- 27:12I/O and EIO Cost problems
- 29:19Optimization solutions
- 30:42Migration lessons learned
- 31:23Results and benefits
- 34:01Q&A
- 44:55Conclusion
related talks
31mA Journey of Migrating Data Platform from Data Center to Cloud
57mBig Data: Threat or Opportunity?
42mBillion-Scale Triumph: Innovating at the Edge of Cloud Data
46mContaining an Elephant: How we moved Hadoop/HBase into Kubernetes and Public Cloud
49mBuilding a 100‑Petabyte Analytics Warehouse
05mHadoop in the cloud: A Nielsen use case
47mHadoop: Lessons Learned
47mSolving Real & Big (Data) Problems Using Hadoop