Tracking every ship in Norwegian waters
summary
Norway ingests and stores every AIS ping — about 100 million AIS messages/day and a 2005–present archive (~40 TB) — into a lakehouse on Azure Databricks with Terraform infra as code. Pipelines turn raw points into voyages, a Maru emissions model (≈330 input variables plus ML to infer missing registry fields), and a ship time‑to‑impact risk score; jobs drop from ~40h to ~4h using serverless clusters. Outputs are versioned data products, open downloads and audit‑logged APIs for municipalities and research.
AI-written summary · about
chapters (15)
- 0:00Introduction to ship tracking
- 3:06Who we are and our mission
- 7:18Why data platforms matter
- 11:32Evolution of data platforms
- 12:53How a data platform works
- 17:55Structuring data in a lake
- 28:45Building our data platform
- 31:17Key data products we created
- 36:36AI in emissions modeling
- 38:26Risk analysis with time to impact
- 40:10Technical details of data processing
- 43:22Optimizing compute with serverless
- 44:03Spatial analysis with H3 hexagons
- 45:56Future roadmap for the platform
- 49:51Conclusion and Q&A
