An Underwater Dance with Data Engineering in Azure
summary
Equinor standardizes decades of messy well‑log data into reusable outputs by parsing LAS/DLIS and other legacy formats, converting to parquet master composites plus JSON lineage, and running the pipeline on Azure Durable Functions with Data Lake and Data Factory. About 120,000 wells (≈6 million log curves, millions of files scanned) move from manual $1k–$3k per well work to about 4 NOK (~$0.40) and ~3 minutes per well on average. Open‑source parsers, Python tooling and unit/name mapping handle units, 17M raw files, and provenance.
AI-written summary · about
chapters (30)
- 0:00Introduction
- 4:36Digital Realm
- 6:22Subsurface
- 8:13Ecuador
- 10:06Nordmark
- 10:58Predictive Business
- 12:19The Challenge
- 13:58What is Well Logging
- 15:16Traditional Well Logging
- 18:08Well Log Data
- 19:43Data Collection
- 21:08Data Science
- 22:02Challenges
- 22:31Units of Measure
- 24:43Data Formats
- 26:03Common Information
- 28:23Why Data Science
- 30:06Challenge 4 Mental Walk
- 31:59Challenges Summary
- 32:23Team
- 33:42Open Source
- 34:42Where to Start
- 36:28Data Complexity
- 38:50Inversion Over Technology
- 41:47Data Engineering Pipeline
- 43:13Data Factory Pipeline
- 44:54Final Products
- 46:02Results
- 48:30Value realization
- 51:20Whats next
related talks
01h 02mData lakes beyond clusters
29mData Acquisition: Overcoming Challenges
18mLessons from Analysing all of Europe's Data
41mEnhancing Event Analysis at Scale: Leveraging Tracking Data in Sports
01h 02mAnalyzing StackExchange with Azure Data Lake
39mWhen Rivers Speak Analyzing Massive Water Quality Datasets
11mBreaking Data Silos: Enterprise Data Management
02h 05mBig Data and the Data Lake from the Analyst Perspective