Lessons from Scaling AI Powered Translation Services Using Istio
summary
Lilt migrates a monolithic translation server onto Kubernetes and uses Istio as API gateway to enable self-serve routing, External Authorization, observability and gradual rollout of microservices; rollout targets ~100 pods across several services and preserves low latency. Intel contributions: the crypto multi-buffer library yields 23–25% latency reduction and ~30% higher QPS, Hyperscan cuts latency ~16% and boosts QPS ~20%, and SGX-based process isolation protects mTLS keys and secrets via attestation — all integrated and proven in production.
AI-written summary · about
chapters (7)
- 0:00Scaling AI translation with Istio
- 3:14Initial architecture and Istio goals
- 6:06Gradual rollout and team strategies
- 8:59Latency versus data security
- 10:22Next steps and optimizations
- 13:55Cultural adoption and future plans
- 17:10Q&A on security and SGX
related talks
24mMigrating a Monolith to Kubernetes with Istio: Our Jou...
28mIstio on Knative Lessons Learned: How Istio is Fit for Serverless Platform
26mIstio Migration: 50k+ Pods, Minimal Impact, Lessons Learned
47mIstio, le meilleur ami de votre cluster K8S
40mIstio, your K8S cluster best friend
28mOptimizing Istio Autoscaling: From Resource-Centric To Connection-Aware
48mIstio An open platform to connect, manage, and secure microservices
50mDeveloping Microservices with Istio