Scaling LLM APIs: Integrating Load Testing Into Your CI/CD Pipeline.
summary
Shows a concrete CI/CD pipeline that packages Gatling simulations and triggers Gatling Enterprise runs from GitHub Actions workflow, using a packaged .sh to deploy tests and optional private load generators across regions (example: 70% France). Tests use feeders (CSV) for varied prompts against LLM endpoints over HTTP/SSE and measure percentiles, errors/sec and trends. Demo launches 5 concurrent users and reports response times from ~100 ms up to >1,200 ms; streaming reduces perceived latency with SSE first-chunk ~0.05s.
AI-written summary · about
chapters (9)
- 0:00LLM API load testing intro
- 1:55What is load testing
- 3:21Server-sent events in APIs
- 4:50Demo streaming LLM responses
- 6:39Gatling load tests setup
- 8:02CI/CD pipeline integration
- 10:30GitHub Actions automation
- 11:53Performance report analysis
- 12:36Resources and support
related talks
34mPerformance tests with Gatling
12mWeb load testing with Gatling
17mDeep dive on Azure Load Testing in CI/CD
30mEfficiently Deploying and Benchmarking LLMs in Kubernetes
31mAIOps: Automating and Scaling ML and LLM Workloads in Production
35mFive Adjustments You Need To Do To Your CI/CD Pipeline Before Integrating LLMs In your Application
23mDistributed performance testing at the push of a button
48mHow to not Crash in Prod, Load Testing Crash Course