BIG DATA STORAGE & PROCESSING: Introduction to Hadoop and Its Ecosystem
summary
Hadoop is presented as a petabyte-scale data storage and processing platform used by Facebook and others, with HDFS storing files in 128–256 MB blocks and a default replication factor of 3. YARN schedules MapReduce and generic containers to place compute near data, ZooKeeper handles master coordination, Hive offers SQL-style access and HBase provides wide‑column storage; Hadoop 3.0 adds erasure coding, federation, and container overcommit to reduce space and scale Namenodes. The talk gives concrete numbers and examples and frames these components and features in terms of architectural tradeoffs and failure tolerance.
AI-written summary · about
chapters (11)
- 0:00Start
- 7:12At the Sources of Big Data
- 10:19Characterization
- 12:46Use Cases
- 14:19Storage Challenges and Big Data Analysis
- 18:07Characteristics of a Distributed Computing System
- 21:59Hadoop and Its Ecosystem
- 23:37The Story of a Plush Elephant
- 36:50HDFS Architecture
- 43:17Anatomy of a Write
- 44:07Anatomy of a Read
related talks
45mFrom Big Data to Big Information
45mSolving Real & Big (Data) Problems Using Hadoop
59mHadoop Essentials for the SQL Server Professional
08mHadoop and Big Data Hype
55mApache HBase: crazy dances on the elephant back
43mMulti-tenant HBase Solutions at Facebook
15mHadoop's Impact on Data Management's Future
49mBuilding a 100‑Petabyte Analytics Warehouse