Miles or Kilometers? Why your data schema should include units
summary
Shows a Scala library, Coulomb, that encodes units as types, compiles conversions, and rejects incompatible units. Recounts a Kubernetes/Spark memory bug where base‑1024 vs base‑1000 mismatch crashed jobs and cites the Mars Climate Orbiter as a costly example. Demonstrates Coulomb Scala library compiling on Scala 2.13, Avro schema integration that stores unit metadata and auto‑converts, and compiler-checked units that stop misconfigured timeouts and invalid API calls from ever building.
AI-written summary · about
chapters (15)
- 0:00Start
- 0:19"Unit" is Overloaded
- 0:44Units of Measure
- 1:04Units from Software
- 1:40Unit Mistake: $327 million
- 3:37What Role Do Units Play?
- 5:03Data Types!
- 5:54Find the Unit Types
- 7:38Units as Data Types
- 8:02Programming With Unit Types
- 8:55Compile-Time Checking
- 9:25Unit Types for Configuration
- 9:46Unit Aware Configuration
- 10:41Unit Aware Avro Schema
- 12:40Review
related talks
38mScaling Scala
39mComplex Data Transformations Made Easy
39m"Unit tests for Data" with Deequ
01h 31mImproving Our Safety With a Modern C++ Quantities and Units Library
28mTHOSE 10000 CLASSES I NEVER WROTE
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
01h 36mSpark for Scala Developers
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka