How Idealo Powers A More Reliable Data Platform With Data Observability
summary
Idealo runs a data platform on AWS with Kafka and Flume ingestion, Presto/Athena query and Monte Carlo for observability, covering nearly 2 billion offers. The team builds a data marketplace with lineage, SLAs and automatic freshness and volume monitors; Monte Carlo caught incomplete batches via a volume anomaly that triggered new downstream checks. Roadmap adds field-health (price anomaly) profiling and a usage-based ranking to assign stricter SLAs, cutting incident triage and resolution time.
AI-written summary · about
chapters (14)
- 0:00Introduction and speakers
- 1:49Data-driven vs data trust
- 4:16Key data use cases at idealo
- 6:24Data platform infrastructure
- 8:09Top data quality issues
- 9:58Data observability program impact
- 12:52Building trust in data
- 15:17Future vision for data trust
- 16:21Roadmap for data platform
- 18:32Monte Carlo pillars importance
- 20:20Data marketplace design
- 21:46Monitor coverage and SLAs
- 24:58Organic data discovery methods
- 27:08Summary and Q&A
related talks
33mDiscover How to Create Your Own Metadata-Driven ML Platform from Scratch
48mFrom (Big) Data Mess to Data as an Innovation Enabler
21mPutting big data self-service analytics in the hands of business people
01h 01mBetter reliability and uptime with analytics and observability
39mCrafting modern data platforms: reveal your data value
36mReal time insights for better products, CX & resilient platform
14mObservability simplified: Empowering engineers to make better software
29mUnifying Real-Time and Data Lake: Yotpo's Transition from Chaos to Coherence