Docker to kubernetes: running Hadoop in a cloud native way

on YouTube ↗41m

" summary

Shows concrete recipes for running Hadoop in containers using docker host network for a Hadoop-native setup and Consul + Nomad for a DIY, service-discovery approach, plus a Kubernetes option using statefulsets and a Prometheus sidecar that scrapes JMX without changing Hadoop. Demonstrations include launcher scripts that render XML from env vars, Consul templates that hot-reload configs, and a sidecar Java agent that exposes metrics — all working prototypes with code and images available.

AI-written summary · about

" chapters (13)

1. 0:00Introduction to talk
2. 1:30Why containerize Hadoop
3. 3:20What is Hadoop and Big Data
4. 5:05Containerization basics
5. 8:15One container vs many containers
6. 11:30Hadoop in Docker Compose
7. 13:39Hadoop with HashiCorp stack
8. 21:12Kubernetes overview
9. 23:21Kubernetes vs Hadoop thinking
10. 25:47Kubernetes ecosystem and flexibility
11. 31:30Final label for Kubernetes
12. 32:35Summary and key takeaways
13. 34:42Q&A session

" related talks

• 53mEffective Kubernetes for Cloud Native Java DevelopersReza Rahman · Developers Summit 2021
• 44mFrom single container to applications at scaleHardy Ferentschik · Øredev 2019
• 24mDynamic Big Data in the Cloud with KubernetesFrank Conrad · Berlin Buzzwords 2019
• 57mReal-World Kubernetes For DevOpsPhil Lombardi · DevOpsDays Boston 2017
• 37mA Hybrid Container Cloud With Kubernetes and Hadoop YARNJian He, Bushuang Gao · KubeCon + CloudNativeCon China 2018
• 25mKubernetes for Java DevelopersArun Gupta · Kubernetes Day India 2019
• 53mBootiful workload orchestration with Hashicorp NomadMartin Ahrer · Spring I/O 2023
• 36mBuilding a Scalable Data Science & Machine Learning cloud using KubernetesMurali Paluru · Big Data Conference Europe 2019