Hadoop at Scale: We Built a Large Cluster, How Do We Maintain It?
summary
Describes operating a large Hadoop cluster after deployment: maintaining reliability, meeting client needs, and preserving the ecosystem at scale. Covers practical lessons and best practices for running Hadoop components and ensuring the cluster continues to serve production workloads.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 4:45Hadoop maintenance problems
- 7:12Infrastructure and growth
- 14:48User control and resources
- 15:13JSON to Parquet migration
- 26:14Resource quotas with rainbow queues
- 31:10Cluster optimization tools
- 34:02Monitoring with Harmadonna
- 36:53Bloom filters for performance
- 39:41Backup strategies
- 42:31Key takeaways
- 43:57Q&A
related talks
47mHadoop: Lessons Learned
44mDeveloping for Big Data at the Intersection of Containerization and Infrastructure as Code
52mOperating OpenSearch at Scale: Lessons From Managing 10,000+ Clusters Across...
35mScaling up to Big Data Devops for Data Science
25mPlatform Engineering for Modern Data Infrastructure
31mSupport Life! Maintaining a Highly Complex Environment
30mManaging Postgres at scale: Challenges, Tools & Techniques
37mManaging Large-Scale Kubernetes Clusters Effectively and Reliably