OpGuard: Bitwise Alignment for Precise and General Debugging of Production LLM Training
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
Claims a practical system, OpGuard, that uses bitwise alignment—lightweight per-op fingerprints plus a schedule-tolerant mapper that finds the longest prefix of bitwise-identical tensors—to localize the first mismatching operator as a concrete debugging pivot. Implements determinism control, preflight stable-op discovery, guarded replay and offline alignment, and runs at production scale at ByteDance. It diagnoses over twenty production issues (kernel races, silent data corruptions, bad machines), cuts triage from days to minutes, and costs ~30% overhead in trusted mode or ~80% for full signatures.
AI-written summary · about
chapters (11)
- 0:00Debugging LLM training
- 1:03Current debugging limits
- 3:54Bitwise alignment insight
- 5:17Practical alignment challenges
- 7:00OpGuard workflow
- 8:07Handling non-determinism
- 9:52Robust alignment across traces
- 12:45Evaluation and deployment
- 14:11Debugging time impact
- 15:35Overhead and scalability
- 16:39Conclusion
related talks
25mRuntimeGuard-AI: Cryptographic Compliance
29mBeyond the Hype: Building Trustworthy and Reliable LLM Applications with Guardrails
21mWhen Oblivious is Not: Attacks against OPAM
38mSecurity for AI Agents Using an Ensemble of Fine-tuned... (Lidan Hazout, Bar Kaduri)
46mGuarding the LLM Galaxy: Security, Privacy, and Guardrails in the...
26mOne guardrail is not enough: Defense-in-depth against LLM jailbreaks
01h 04mRobust Generalization in the Era of LLMs: Jailbreaking Attacks and Defenses
01h 00mAI Safety through Programming?