The Future of Inference for Complex AI Applications
summary
Claims a working, stateless synchronous DAG executor — an inference graph execution service — that stitches ONNX/Triton model calls, feature-store fetches, vector DB lookups and FaaS nodes into a single protobuf/gRPC API with per-node types, metrics, timeouts and a Python DSL to compile graph configs. Implements async, future-style concurrency and plugin node schemas; used on production-style chatbot/recommender/search stacks at Salesforce and targets typical end-to-end latencies of 50–500 ms, interoperable with Ray Serve and Kubeflow.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:12Multi-stage inference architecture
- 3:59Synchronous inference
- 6:49Inference categories
- 8:15Synchronous DAG execution
- 10:25Inference graph service
- 13:58Node types and schema
- 16:03Input data handling
- 17:05DAG configuration generation
- 18:30Conclusion
related talks
19mAI Inference Pipelines – Building Low-Latency Systems With gRPC
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
35mAdvanced Model Inferencing Leveraging KNative, Istio & Kubeflow Serving
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)
13mTailor-Made Inference at Scale
25mSponsored Session: Accelerating GenAI Inference: From AWS Deep Learning...
21mXorbits Inference: Model Serving Made Easy