How to Process 80 Million Events a Day and Build Relational Graphs with Postgres
summary
Processes 80 million daily events into relational graphs using Amazon Kinesis and Lambda feeding Aurora Postgres. Tests showed dedicated graph DBs (Amazon Neptune) were slow for many small shallow graphs, so an edge/relations adjacency-list (triple-style) table was used; race conditions at ~1k concurrent writers were handled by immediately writing relations, batching root updates in 5-minute windows, scaling reads with replicas, and offloading failed events to S3 for manual reprocessing, achieving near-real-time scale.
AI-written summary · about
chapters (11)
- 0:00Event processing and graph building
- 4:21Event-based benefits
- 10:58Graph database challenges
- 14:12Relational hierarchies
- 19:17Lambda architecture
- 22:15AWS Lambda and Kinesis
- 25:08Race conditions in rebuilds
- 28:21Scaling to 80M events
- 31:37Adjacency list performance
- 33:24Final architecture
- 36:59Scaling lessons
related talks
20mLessons learned scaling our SaaS on Postgres to 8+ billion events
43mJust Use Postgres.
34mAnalytics and AI with PostgreSQL - Mission Possible!
28mUsing a database in a data science project - Lessons learned in production
51mThe Mother of All Databases
48mBuilding Petabyte-Scale PostgreSQL Deployments
55mBreakout Session
34mAnalytics and AI with PostgreSQL - Mission Possible!