Migrating 20K nodes and scaling to production on EKS & MSK in 8 months
summary
New Relic migrates over 20,000 servers in eight months, refactors from self‑managed EC2 Kubernetes to Amazon EKS, and shifts Kafka to MSK with about 40 MSK clusters to gain fault isolation. The platform ingests 120 PB/month and >1B datapoints/min into NRDB while running 19,000+ EKS nodes and 150,000+ pods. Migration uses a smart HTTP routing layer, lift‑and‑shift then refactor approach, heavy automation (Terraform, Kubernetes), self‑service topic tooling and Cruise Control to balance clusters.
AI-written summary · about
chapters (28)
- 0:00Introduction and speaker intro
- 0:30Migration journey overview
- 1:14Scale challenges at New Relic
- 3:57Current architecture deep dive
- 4:59Risks of legacy infrastructure
- 6:26Why choose AWS Managed Services
- 7:29Write-heavy workloads explained
- 8:14Migration strategy and planning
- 9:21Isolated environments in AWS
- 10:27Tick-tock model for innovation
- 11:10EKS lift and shift approach
- 11:32MSK modernize first approach
- 12:16EKS optimization and scaling
- 12:40Sherbert chart migration progress
- 13:24De-risking with observability
- 14:31Instrumenting Kubernetes workloads
- 15:16Managing migration risks
- 15:59Daily migration tracking views
- 16:42Real-time issue detection
- 17:28Drinking own champagne benefits
- 18:12NR1 full stack observability
- 19:16Q&A with AWS partners
- 19:58Current migration status update
- 21:05MSK benefits and lessons learned
- 22:30EKS benefits and certifications
- 25:52EKS reliability and security gains
- 27:22Anomaly detection in NR1
- 28:47Closing remarks and survey
related talks
36mMigrating from kOps to EKS with Minimal Developer Involvement
33mNavigating the Seas of Transformation: Migrating from EC2 to Kubernetes
49mHardware to AWS, EC2 to EKS: Two Years, Two Epic Migrations
09mHow migrating from ECS to EKS supercharged our infrastructure
29mFrom EC2 to Kubernetes Takeaways from a Brownfield Migration
14mMigrating and running production services in Kubernetes
23mParadox's EKS Migration
46mMigrating from VMs to K8s