Evolving and Supporting Stateful, Multi-Tenant Decisioning Applications in Production

on YouTube ↗35m

" summary

Kubernetes runs a stateful, multi-tenant decisioning platform that stitches Apache NiFi for ETL, Kafka as the persistent stream backbone, Flink for real‑time decisioning, Pachyderm for versioned-data model refits, and Drill/Zeppelin for ad‑hoc analytics. Cluster automations enforce a 60‑day rehydration, PersistentVolumes + StatefulSets handle state, and resource quotas plus pod affinities prevent noisy neighbors. Production workloads process ~6 million transactions/day, retain 180 days of stream data, and currently use a shared Flink cluster with a CLI/CRD path toward per‑tenant clusters.

AI-written summary · about

" chapters (25)

1. 0:00Intro
2. 0:48Welcome to balmy Austin!
3. 2:12Our experience with K8s
4. 5:10Production Workloads
5. 9:26T1: Decisioning Engine
6. 10:25Model Refit with Pachyderm
7. 11:04Analytical Environment
8. 12:39Telemetry: Responsive Reactive
9. 13:21Def. State and Multi-Tenancy
10. 14:49Value to customers - a managed service
11. 15:45So you want to be a platform provider? Customer interactions to expect
12. 16:37"Well-behaved" kes workloads
13. 18:02Your experience?
14. 18:57Tenant Isolation - Namespacing
15. 19:47Stateful Applications & Pod-Volume Affinity
16. 21:10Sidebar: Automated Upgrades & "Rehydration"
17. 22:27Lessons Learned: Safety First!
18. 24:05Future state: Elastic/Dynamic Load
19. 24:45Kubernetes should be invisible
20. 25:59Current User/Platform Interaction
21. 27:45Learning Opportunities
22. 28:33Future User/Platform Interaction
23. 30:42K8s Architecture: C.f. In-flight drone maintenance
24. 32:22Conclusions
25. 34:42Community Shout-out!

" related talks

• 26mKubernetes & real-time analytics: How to connect these worlds with Apache FlinkAlbert Lewandowski · Berlin Buzzwords 2021
• 16mZero-downtime upgrades of Flink applicationsLakshmi Rao · Flink Forward Global Virtual 2020
• 36mKubernetes, DecisionsKobi Biton · SREday 2023 London
• 34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data MeshRakesh Subramanian Suresh, Jainik Vora · Data on Kubernetes Day North America 2023
• 24mDynamic Big Data in the Cloud with KubernetesFrank Conrad · Berlin Buzzwords 2019
• 05mLightning Talk: What Makes a Good Multi-tenant Kubernetes Solution?Victor Varza · KubeCon + CloudNativeCon North America 2019
• 58mReal-Time ML Applications at Scale with Flink and Beam on Kubernetes and GCPChethan U · Flink Forward Global Virtual 2020
• 37mSpark and Flink Running Scalable in Kubernetes #bbuzzFrank Conrad · Berlin Buzzwords 2018