Cloud Assumes You Know What a Request Will Cost
summary
Compares two executions of the same agent prompt—investigate a checkout latency spike—and shows a clean path with 5 nodes versus a failed-test path with 7 nodes, where the second run costs meaningfully more despite returning a 200. Explains why timeouts, capacity, rate limits and per-request billing break when work is discovered mid-run, recommends giving every run a per-run budget (tokens/dollars) enforced inside, plotting P95/P99 not the mean, and splitting workflows and sandboxes for execution and environment.
AI-written summary · about
chapters (39)
- 0:00Cloud Assumes You Know What a Request Will Cost
- 1:30The Assumption Agent Workloads Break at the Source
- 2:02Speaker Intro - Ojus Save, Developer Relations Engineer, Render
- 3:03Every Piece of Cloud Infrastructure Makes One Assumption Before Your Code Runs
- 3:33Timeouts, Capacity, Rate Limits and Billing All Rest on the Same Belief
- 4:37Two Runs of the Same Instruction, One Costs Meaningfully More
- 5:08The Whole Problem in One Picture: One Request, Many Decisions
- 5:40The Thing You Count and the Thing That Spends Money Are Different Objects
- 6:12Naming the Admission Event and Its Four Commitments
- 7:13What the System Actually Knows at That Instant
- 7:43Not Hard to Estimate - Undefined
- 8:13The Cruel Part: It Returns a 200 and Nothing Tells You the Guess Was Wrong
- 8:43An Audience Bet: How Many Steps Will This Instruction Take?
- 10:44Run One - The Clean Path: Five Nodes
- 11:46Run Two - Same Prompt, the Test Fails, Seven Nodes
- 12:47This Is a Completely Successful Run, and It Still Costs More
- 13:21Node Count Is Misleading - Context Drags Along and Grows
- 14:54Which Run Was Your Timeout Sized For?
- 15:24The Argument: A Request Is the Wrong Thing to Price and Bound
- 16:27What You Want Is the Run and Its Budget
- 17:32Three Infrastructure Problems That Fall Out
- 18:04Problem One - Compute: A Full Pool With No Traffic Spike
- 18:36Why Averages Fail: Agent Cost Has No Thin Tail
- 19:38Provision for the Mean and You Queue; Provision for the Worst and You Waste
- 20:09Go Plot Your Distribution - Look at P95 and P99, Not the Mean
- 21:10The Boundary Case Where Failure Becomes Invisible
- 21:42A Timeout Sized for a Normal Request Firing Mid-Run
- 22:13A Plausible Summary of Work That Did Not Finish
- 22:46You Paid for All of It and the Accounting Recorded a Success
- 23:20Separate Per-Tool-Call Timeouts From Per-Run Budgets
- 23:50Problem Two - Environment: Scheduling a Task Isn't Giving It Somewhere to Run
- 24:53Isolated and Disposable, Pulling Against Each Other
- 25:23Problem Three - Context: Three Layers People Constantly Conflate
- 25:53A Larger Context Window Is a Cache, Not a System of Record
- 26:56Three Questions: What Can Be Reconstructed, What Must Be Durable, What May Continue
- 27:58Why Per-Request Rate Limits Do Not Bind Agent Workloads
- 28:29Give Every Run a Budget at Admission and Enforce It Inside
- 29:33Two Primitives: Workflows Own Execution, Sandboxes Own Environment
- 33:22Q&A: How to Validate Agentic AI Workflows
related talks
20mCertainty Among the Chaos
40mCloud Costs Panel "Lots of Reward Points"
10m{ Fast, Cost-effective } – Choose Two
35mAzure SpendOps – The Art of Effectively Managing Azure Costs
21mDon't Become Your Agent's B****
34mOptimizing Cloud Efficiency with Prompt Engineering
25mYOUR AI DOESN'T KNOW WHAT THINGS COST
33mKubernetes cost optimization - How to actually do it