Brett Rudenstein - BigDataNYC 2013 - theCUBE - #BigDataNYC
summary
Demonstrates a single active-active HDFS spanning two EC2 regions (Northern Virginia and Oregon, about 3,000 miles apart) running teragen/terasort MapReduce to show synchronous local writes with asynchronous “foreign blocks” replication. Three active namenodes per site (six-node cluster) handle client round-robin requests, survive a local namenode reboot and a remote namenode kill, tolerate a full WAN partition, then self-heal and resume block replication so jobs continue with essentially zero failover and consistent global metadata.
AI-written summary · about
chapters (14)
- 0:00Introduction and setup
- 1:03Non-Stop Hadoop overview
- 2:06Name node explained
- 3:09Active active replication
- 4:54Share nothing architecture
- 7:40Business value and use cases
- 10:50Competitive comparison
- 11:51Demo introduction
- 12:12Running TeraGen and TeraSort
- 13:38Failing name nodes during job
- 14:43Self-healing process
- 16:28WAN link failure test
- 17:52Comparison to other resiliency methods
- 19:15Product development challenges
