How to Survive to High Demanding Services
Talkat Open Source Summit Europe 2023 · on Wednesday, 2023-09-20· at 12:20· Room 0A (Floor 0), Abandoibarra Etorb., 4, 48011 Bilbao, Bizkaia, Spain·
summary
AI translation and TTS services run on GPUs in Kubernetes and absorb ~400,000 requests in 12 hours from ~30,000 IPs. Logs use a custom Fluentd sidecar sending logs to Datadog (custom image embeds Node scripts so developers format parsers), dashboards power Datadog alerts forwarded to Microsoft Teams, and synthetic HTTP/navigation checks fire alarms. Traffic protection uses Istio rate limit with API token+clientIP+service+path (per-second and per-minute Redis backends split for performance). Canary and shadow deployments use virtual services and Jenkins pipelines to safely test versions and return 429 on overlimit.
AI-written summary · about
chapters (7)
- 0:00Surviving high demand services
- 2:06Monitoring service metrics
- 9:26Real-time error detection
- 13:18Denial of service protection
- 20:18Shadow deployments
- 22:39Canary releases
- 32:22Summary
related talks
34mHow to Survive to High Demanding Services
14mMonitoring Network Traffic in Kubernetes
41mKubernetes the Very Hard Way
11mK8s-Native AI Microservices: Scaling in Regulated Industries
44mKubernetes at Datadog Scale
27mHarnessing Kubernetes for Scalable AI/ML Workloads
01h 41mDatadog DASH 2019 Keynote
24mSleeping soundly even when scaling your app to 200k request/seconds