Developing for Big Data at the Intersection of Containerization and Infrastructure as Code
summary
Claims a repeatable path to let teams self‑provision Apache Hive on Hadoop by first building local containers then lifting to cloud, using docker compose for five‑container dev setups and terraform to provision eight EC2 instances (one name node, four data nodes, one Hive RDBMS, metastore, server). Includes a Python CSV cleanup, Beeline commands, a MapReduce wordcount smoke test that passes locally and on AWS, notes a 10‑minute transform and 3‑minute download, and shows teardown removes volumes and instances.
AI-written summary · about
chapters (8)
- 0:00Start
- 3:17Richard Feynman
- 23:40Put Together a Schema File
- 30:48Smoke Test Script
- 33:50Provisioning the Rdbms
- 39:40Performance Graph
- 40:16Takeaways
- 42:12How Do You Learn
related talks
35mScaling up to Big Data Devops for Data Science
59mHadoop Essentials for the SQL Server Professional
28mDevOps in a Data Warehouse: Inside Out
49mBuilding a 100‑Petabyte Analytics Warehouse
07mComplementary Solutions with Hadoop
46mDeploying and maintaining software for today's infrastructure
45mSolving Real & Big (Data) Problems Using Hadoop
44mAutomating a Big Data Platform with Jenkins