Real-time Energy Trading: Alpiq's Strategic Data Platform on AWS
summary
Alpiq describes a production data platform on AWS that ingests 3 billion transactions/day (≈2 PB) from 30 source types, unifying market, grid and battery streams into S3/iceberg with Snowflake and dbt for analytics. They run MSK serverless + Amazon Managed Flink for streaming, EKS/Fargate for ingestion, Athena/QuickSight and APIs for consumption, and Bedrock/QuickSight Q for schema automation. Reusable components and CI/CD cut time-to-delivery 60% faster and reduce infrastructure spend 75% lower infra costs, proving the platform scales and serves real-time trading. Snowflake + dbt MSK serverless + Amazon Managed Flink
AI-written summary · about
chapters (23)
- 0:00Introduction and speakers
- 1:05Who is Alpiq and our business
- 3:39Challenges we faced
- 7:37Why we built a data platform
- 8:22Data mesh principles
- 9:50Platform evolution and pillars
- 11:16Platform architecture overview
- 13:03Technology stack and integration
- 14:28Team structure and operations
- 15:31Developer and consumer workflow
- 16:56Battery energy storage systems
- 18:43BESS Day One use case
- 21:58ELT pipeline implementation
- 24:25BESS Day Two streaming case
- 26:09Streaming analytics fundamentals
- 28:17BESS Day Two architecture
- 32:14Sales and AI use case
- 33:39Grid mapping challenges
- 35:02AI-driven schema management
- 37:53Reusable components and AI
- 38:14Business case results
- 39:19Challenges and solutions summary
- 40:45Conclusion and future outlook
related talks
16mHow PCI Energy optimizes energy trading wth cloud-based solutions
10mBuilding a Data Platform at Assembly Payments
03mStartup Showcase: Alpine Data Labs
33mAnalytics at scale: Migrating Intuit to AWS with cloud-native tools
29mSimplifying data flows
29mData foundation for analytics and AI
27mHow To Develop & Operate An Agile Cloud Datawarehouse In A Rapidly Changing Multi-ERP Environment
29mHow financial services companies derive greater value from data with AWS