Inference on Streaming Data at Scale at Intuit
Talkat Cloud Native + Kubernetes AI Day NA 2024 · on Tuesday, 2024-11-12· at 15:55· Salt Palace | Level 1 | Grand Ballroom A·
summary
Numaflow is a Kubernetes-native open-source platform that claims to let ML teams plug Python/Java/Go/Rust UDFs into pipelines that read from Kafka, Pulsar, SQS or HTTP, run inference (including GPU placement for heavy models), and republish results; a deployed Intuit instance runs millions of model jobs and about 40 million inferences a day. Autoscaling uses vertex buffer depth and Kafka ack-on-write so pipelines scale down to zero and then rebound; demos show LLM image captioning and diffusion image generation working end-to-end, and pipelines can trigger training or route to different models conditionally.
AI-written summary · about
chapters (10)
- 0:00Streaming inference intro
- 1:25Stream processing challenges
- 3:32Numa Flow solves it
- 5:15Numa Flow architecture
- 6:18Live image recognition demo
- 11:52LLM request-response
- 13:17Multi-model pipeline
- 15:43Production pipeline at Intuit
- 17:30Numa Flow features
- 18:53Q&A
related talks
19mHow Intuit Streamlined AI/ML Inference Workflows on K8s
29mSimplify Stream Data Processing with Declarative, Scalable K8s Native Technologies
23mAnalyzing Operational Data at Scale Using ML at Intuit
28mAccelerating AI at Scale with Data
40mHow to leverage the Apache Kafka Ecosystem to productionize Machine Learning
35mAdvanced Model Inferencing Leveraging KNative, Istio & Kubeflow Serving
28mNo JVM, no Kafka: the future of streaming data pipelines
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow