Closing Gaps From Commercial Tools in German Redisp...
summary
A lightweight script grows into a companywide data hub that ingests about 6 million measures and 15 GB of raw input daily from 20+ systems, reconciles moving master data for 150,000 plants, and applies automatic corrections, cost forecasting and risk assessment so settlement errors surface before payments. Architecture uses Delta Lake and Polars for storage, fast hashing and row versioning, cutting a 1M-row 144-column hash from 5 minutes to ~20 seconds and enabling wider reuse across departments.
AI-written summary · about
chapters (21)
- 0:00Introduction and speakers
- 1:03Who is Bayernwerk and redispatch
- 3:56Growth of curtailment in Bavaria
- 5:01Redispatch 2.0 process explained
- 6:31Challenges in redispatch process
- 9:04Biggest problem in redispatch
- 10:08Development of REP tool
- 12:13Measuring redispatch measures
- 14:07REP tool functionalities
- 15:53Cost forecasting and data gaps
- 16:58IT and OT system integration
- 18:02Open source vision for REP
- 19:06Systems involved in redispatch
- 20:12Data ingestion and correction
- 22:26Single source of truth limitations
- 23:30Data volume and growth
- 24:31Technical architecture of REP
- 26:41Metadata and performance
- 27:23New use cases with REP
- 28:27Future plans and cooperation
- 28:55Q&A session
related talks
41mIncremental Architecture
37mData Swamp to Data Swimming Pool: Cleaning up the Mess
43mArchitecting for Speed & Scale
01h 02mShift Data Quality Left with Data Mesh Principles
38mLesson learned implementing a large-scale data mesh transformation
42mHow Data & Software Eng. Teams Collaborate to Ensure Smooth Data Integrations
01h 35mRedesigning Legacy Systems - Keys to Success / Lessons Learned
57mPatterns of SaaS: Refactoring Databases