Lessons learned scaling our SaaS on Postgres to 8+ billion events

on YouTube ↗20m

" summary

ConvertFlow scales from Rails on Heroku Postgres to a Citus sharded cluster on Azure, processing 8.7 billion events and serving ~40M monthly visitors; switching off Firebase cuts cloud costs. Shard analytics by tenant ID with Citus (Hyperscale) to get 2–4x faster queries, use rollup tables and daily increments to reduce query work, reindex CONCURRENTLY to avoid locks, and perform a bigint migration for primary keys with a short maintenance window.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 0:58What is Convertflow
3. 2:27From 0 to 1M events
4. 6:42From 1M to 100M events
5. 8:57From 100M to 1B events
6. 12:13From 1B to multiple billion events
7. 15:03Bigint migration
8. 16:08Migration to Hyperscale (Citus)
9. 17:19Lessons learned recap

" related talks

• 52mScaling a Saas application beyond a single Postgres with CitusJesse H. Willett · PostgresOpen Silicon Valley 2017
• 24mCitus from the Customer eyesSai Krishna Srirampur · Citus Con: An Event for Postgres 2023
• 16mOnline Scale-Out and Cloud Data IngestionAdam Wolk · Azure Cosmos DB: Liftoff 2022
• 24mLessons learned: Migrating from AWS-Hosted PostgreSQL RDS to Self-Hosted CitusMatt Klein · Citus Con: An Event for Postgres 2023
• 55mBreakout SessionDan Robinson · PGConf Silicon Valley 2015
• 32mAuto scaling Azure Cosmos DB for PostgreSQL with Citus, Grafana & Azure ServerlessLucas Borges · Citus Con: An Event for Postgres 2023
• 02h 02mAzure Cosmos DB: LiftoffKirill Gavrylyuk, Estefani Arroyo, Iria Osara, Sai Krishna Srirampur, Claire Giordano · Azure Cosmos DB: Liftoff 2022
• 16mKeynote – Umur CubukcuUmur Cubukcu · PGConf Silicon Valley 2015