When Agents Meet Physical Data: The Other Physics of Agent Harnesses
summary
Claim: recompute is the enemy for video and sensor data — materialize once, recall cheaply. Demonstrations use Claude Code over raw S3 with a YOLO pass that processes 91 dashcam clips in ~24 minutes and yields ~100,000 records so follow-ups answer in seconds and cents instead of reprocessing; a deployed system needed layers to operate over 600 petabytes and shows measured gaps on the order of millions between recompute and recall. Implementation specifics: Pydantic schemas, execution engine, incremental checkpoints, and a knowledge base that makes dataset the unit of state.
AI-written summary · about
chapters (13)
- 0:00Physical data challenges
- 1:50Agents and unstructured data
- 3:34Video data object explosion
- 5:41Schema complexity with Pydantic
- 7:10Dashcam video analysis demo
- 13:27Schemas and data structure
- 14:54Execution engines for physical data
- 18:29Efficient updates and checkpointing
- 19:34Testing in data projects
- 21:41Metadata with star schema
- 23:06Knowledge bases for harnesses
- 25:58Data harness stack assembly
- 27:03Conclusion and project overview
related talks
25mFrom SQL to Python: Building Data Context for Agents and people
23mYour Agent Has Amnesia: Why Mobile AI Agents Need Memory to Be Useful
30mBeyond GPUs: Why AI Agents Need a Need New Hardware
40mAgentic Retrieval: Building Self-Optimizing Search Systems
35mThe Fight for the Future of Artificial Intelligence
25mWhy Eval++ Is the Next Great Compute Primitive
07h 47mAI Agents Conference 2026 - Day 5
30mBuilding a Self-Service Platform for Stateful AI Agents