The Limits of Safety Engineering and the Cinderblock Problem
summary
Argues that safety engineering has hard limits: it proves positive requirements through hazard assessment, design mitigation and implementation assurance but cannot cover deliberate or wildly unforeseen attacks — the coined Cinderblock Problem — an arms race that forces overdesign. Uses examples (a sudden truck fire, Tacoma Narrows, Challenger O‑ring, Chernobyl, MCAS) to show unknown unknowns and failures of hazard assessment or assurance. Recommends treating adversarial issues with security practices, lower attack surface, and use terms like operate as designed and trusted boot instead of “secure.”
AI-written summary · about
chapters (37)
- 0:00Introduction
- 1:24Safety engineering overview
- 2:26Software bugs intuition
- 3:31AI image bias
- 4:33Bugs versus defects
- 5:34Professionalizing safety
- 6:17Defects vs regular bugs
- 7:19System of systems
- 8:00Best practices limits
- 9:06Operational assumptions
- 10:12Three pillars of safety
- 10:54Hazard assessment limits
- 11:56Tacoma Narrows failure
- 12:39Unknown unknowns
- 12:59Design mitigation challenges
- 13:43Chernobyl design flaws
- 14:30Testing limits
- 15:15Challenger seal failure
- 15:59Operational assumptions
- 17:23Cinderblock problem
- 18:30Satisfiability arms race
- 19:11Why cinderblock persists
- 20:37Security role in cinderblock
- 21:20Safety vs security
- 22:02Proving positives vs negatives
- 22:42Attack surface reduction
- 23:23Stop saying secure computing
- 23:44Trusted boot chains
- 24:34Q&A
- 24:56DO-356A discussion
- 27:56Safety vs security boundaries
- 29:00Learning from disasters
- 30:03MCAS hazard assessment
- 31:30Management pressure risks
- 33:16Moral responsibility
- 34:19Engineer accountability
- 34:39Closing remarks
related talks
46mA Systems Approach to Safety and Cybersecurity
28mWith great power comes great responsibility
01h 28mThe Unfalsifiability of Security Claims
58mSafety-critical systems from the inside
50mWhen Human Life Depends on Software - Introduction to Safety-Critical Systems
56mBeyond the Known Knowns
42mThe History of Fire Escapes
48mEngineering You