Scaling to Billions of Internet Observations per Day
summary
Censys moves its ingestion from Postgres to YugabyteDB to handle roughly 15 billion IOPS per day and an inventory of ~670 million services, using Kubernetes on GKE with n1‑highcpu‑16 nodes and four local SSDs. The team implements CQRS, application‑level sharding to avoid write hot spots, and split queries to avoid tombstone-heavy scans. Per-node performance hits ~300 SQL QPS and ~1,500 CQL QPS (40–100 ms latencies); backups and statefulset updates require custom operational work.
AI-written summary · about
chapters (10)
- 0:00Start
- 0:28Background and Use Case
- 1:14System Requirements & Why Traditional SQL Doesn't Cut It
- 3:01Scaling Up
- 3:55Monotonically Increasing Range Indexes
- 8:17CQRS with YugaByte
- 12:05Takeaways
- 13:32Choosing a Kubernetes Deployment
- 15:01How We Deployed YugaByteDB in Production
- 18:26Operational Lessons Learned and Things to Remember
related talks
30mFrom monolith to petabyte scale: Built a cloud-native, AI-ready data layer
18mBuilding a Scalable, Resilient System for Real-Time Fraud Detection
19mHow Admiral Scales YugabyteDB Effortlessly on Google Cloud
24mCitus from the Customer eyes
20mLessons learned scaling our SaaS on Postgres to 8+ billion events
25mServerless Osquery Backend and Big Data Exploration
22mIt’s all about the db - a journey to accommodate 100k events per second
47mScaling: from 0 to 25 million users