Kubernetes & Data Engineering
summary
Explains why Kubernetes has become important for data engineering, pointing out that container portability plus Kubernetes’ robustness and scalability make it useful for modern data platforms. Describes how Kubernetes (K8s) can be applied to a datalake or lakehouse architecture.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:41Big data governance
- 1:44Data processing containers
- 3:08Microservices & data architecture
- 4:35Governance popularity
- 6:20Auto-scaling & health
- 7:27CI/CD & parity
- 8:52Image & container best practices
- 10:21Security & namespaces
- 11:49HDFS performance config
- 12:32ML & pipeline tools
- 14:58Governance costs
- 16:47Monitoring & stress testing
- 17:31Q&A
- 18:34Governance on Windows/Linux
- 20:00DataOps & quality
- 21:26Getting started governance
- 22:50Summary
related talks
28mProgress For Big Data In Kubernetes
47mData Mesh in Kubernetes
40mData Engineering in a Rapidly Changing World
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
05mHow to Simplify Data Scientist’s Day in Huge Company Using Kubernetes
04mOpening Remarks
29mKubernetes your life insurance for the cloud
06mHow to Make Kubernetes Smarter