Hadoop: Lessons Learned

on YouTube ↗47m

" summary

A practical rundown of running Hadoop in production: a 20-node grep vs MapReduce test (1.5 GB per node) shows MapReduce overhead often outweighs benefit; an ETL on Amazon Elastic MapReduce cuts a migration from two weeks to two days; choose snappy compression for CPU/IO balance; prefer many reducers, use combiners and partitioners to cut IO, watch small-file pressure on the NameNode and append/record-reader quirks, and serialize with proto buffers for faster parsing — use Hadoop only if you must.

AI-written summary · about

" chapters (29)

1. 0:00Intro
2. 1:20Why Hadoop
3. 4:57HDFS
4. 7:13Benchmarks
5. 14:35Integration
6. 22:15MapReduce
7. 25:00Output
8. 27:35Pipeline
9. 28:50Append
10. 29:25Serialization
11. 30:30Map Writable
12. 32:00GetBytes
13. 32:50Iterator
14. 33:45Leucine
15. 34:25Data Structures
16. 35:15Use many reducers
17. 36:30Operations
18. 37:00Hardware configuration
19. 37:15Monitoring
20. 38:27snappy
21. 39:10Hadoop expired
22. 40:25One major bug
23. 41:15Record weirdness
24. 42:30Replication factor
25. 42:50Community
26. 43:20Enterprise effect
27. 44:40Incremental realtime
28. 44:50Other projects
29. 46:15Takeaway

" related talks

• 10mSeparating Hadoop Myths from RealityJack Norris · Strata Conference New York + Hadoop World 2013
• 29mBig Data AnalysisSerhiy Masyutin · JavaDay Lviv 2015
• 08mHadoop and Big Data HypeJack Norris · Strata Conference New York + Hadoop World 2013
• 34mAvoiding Big Data AntipatternsAlex Holmes · JavaOne 2015
• 10mMapReduce in 10 MinutesEvan Smith · PyOhio 2020
• 45mHadoop in the Cloud: The Story of Migrating Hundreds of PetabytesMikhail Maryufich · HighLoad++ 2023
• 47mSolving Real & Big (Data) Problems Using HadoopEva Andreasson · GOTO Chicago 2014
• 56mResistance is Futile: What You Need to Know about Big DataMichelle Ufford · PASS Business Analytics Conference 2015