Understanding French Behavior, It's Possible
summary
LebonObservatoire processes 29 million monthly users and ~500,000 daily postings to detect demand and price trends across 64 categories by ingesting ~15 TB and aggregating daily into monthly metrics. The pipeline uses an S3 data lake orchestrated with Airflow, Citus/Postgres with columnar storage and HyperLogLog for fast unique counts, and applies a max‑p style geo‑clustering with INSEE features plus Facebook Prophet to detect seasonality and event-driven trends; APIs serve department- and commune-level results with typical responses under 200 ms.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 5:20Project architecture & tech
- 13:04Data modeling by category
- 20:29PostgreSQL performance tuning
- 29:16Geo-clustering for data deserts
- 35:14Trend detection with NLP
- 39:08Project feedback & philosophy
- 41:14Q&A
related talks
18mLessons from Analysing all of Europe's Data
43mData Acquisition Made Easy
45mData as culture as software
38mData platforms synchronisation journey at Carrefour
38mCrunching data in real time for Fortune500 clients
22mSupercharge your app with Firebase and BigQuery
22mGenerating a Billion Personal News Feeds
17mIt's All in the Cloud