The Algorithm That Nearly Killed Me: When Testing Isn't Enough
summary
An insulin pump system where the control algorithm on the phone confidently reports high insulin delivery while the hardware actually sends zero units causes blood sugar to climb into dangerous ketones and near diabetic ketoacidosis in 72 hours. The app’s learning amplifies bad training data; the device defaults to 1 unit/hour when disconnected and some models dose 0.1 units per hour as a safety drip — small numbers with large effects. Tests show green ticks but the real system fails; validation must check decision boundaries, invariants and overlap of edge cases.
AI-written summary · about
