Resistance is Futile: What You Need to Know about Big Data
summary
Asserts that most teams will need Hadoop soon and describes a production data lake holding petabytes with 88,000 catalog objects, where 80–90% of company data is unstructured. Shows tools and flow: HDFS, Hive, Pig, Hortonworks, Sqoop, Hue, Python, Tableau and MPP vs SMP performance — same massive joins drop from hours to under a minute on MPP. Describes using Hadoop as a data refinery to parse dirty third‑party files in 30 minutes, and cites UX, tooling and Linux familiarity as the main adoption obstacles.
AI-written summary · about
chapters (11)
- 0:00Big data and Hadoop intro
- 5:59Unstructured data explosion
- 16:54Why Hadoop is essential
- 23:21How MPP systems work
- 28:21Hadoop ecosystem explained
- 31:57Hadoop with existing systems
- 37:01Hadoop as data refinery
- 41:31Querying semi-structured data
- 46:49Hadoop for analytics
- 50:02Hadoop challenges
- 53:45Q&A
