The Limits of Safety Engineering and the Cinderblock Problem
summary
Defines the Cinderblock Problem as attacks or unforeseeable misuse that force an unwinnable arms race if treated by safety methods; breaks safety into hazard assessment, design mitigation, implementation assurance and shows their limits with real failures (Tacoma Narrows, Challenger O‑ring, Chernobyl, MCAS). Argues safety must assume systems operate as designed and are maintained as specified, while security is a separate, ongoing discipline focused on what shall not happen, so minimize attack surface and use trusted computing primitives rather than claiming something is “secure.”
AI-written summary · about
chapters (37)
- 0:00Introduction
- 1:24Safety engineering overview
- 2:26Software bugs intuition
- 3:31AI image bias
- 4:33Bugs versus defects
- 5:34Professionalizing safety
- 6:17Defects vs regular bugs
- 7:19System of systems
- 8:00Best practices limits
- 9:06Operational assumptions
- 10:12Three pillars of safety
- 10:54Hazard assessment limits
- 11:56Tacoma Narrows failure
- 12:39Unknown unknowns
- 12:59Design mitigation challenges
- 13:43Chernobyl design flaws
- 14:30Testing limits
- 15:15Challenger seal failure
- 15:59Operational assumptions
- 17:23Cinderblock problem
- 18:30Satisfiability arms race
- 19:11Why cinderblock persists
- 20:37Security role in cinderblock
- 21:20Safety vs security
- 22:02Proving positives vs negatives
- 22:42Attack surface reduction
- 23:23Stop saying secure computing
- 23:44Trusted boot chains
- 24:34Q&A
- 24:56DO-356A discussion
- 27:56Safety vs security boundaries
- 29:00Learning from disasters
- 30:03MCAS hazard assessment
- 31:30Management pressure risks
- 33:16Moral responsibility
- 34:19Engineer accountability
- 34:39Closing remarks
related talks
46mA Systems Approach to Safety and Cybersecurity
16mDesigning Holistic Security Programs
53mOptimistic Security
35mMove fast, build things… Safely!
21mSpecial Keynote
01h 28mThe Unfalsifiability of Security Claims
01h 00mApplying Security Engineering Principles to Complex Composite Systems
24mGreat Security Is One Question Away