How Similarweb Serves 100s of TBs to Their Worldwide Users in Milliseconds
summary
Similarweb serves hundreds of TBs of web data with consistent millisecond latency by combining key‑value fast paths for simple time‑series with a columnar query engine (Firebolt) for complex analytics, storing ORC/Parquet on S3 and using compute/storage separation plus engine warm caches on SSD. A 72 TB keywords table returns a complex keyword query by scanning only 80 MB and returning results in 70 ms response while bulk ingest uses DynamoDB and managed import to cut cost.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 3:04Key-value vs SQL engines
- 5:16DynamoDB pricing and use cases
- 8:33DynamoDB limitations and workarounds
- 10:44Introducing Presto and compute-storage separation
- 13:37Presto performance and scalability
- 15:52Indexing strategies in Presto
- 17:23Q&A and summary
related talks
15m100x faster data app experiences delivered cost-effectively
41mAnalyzing Terabytes of Data with Google BigQuery
29mCatching Waves with Time-Series Data
36mSearch on the fly: how to lighten your Big Data
30mCatching Waves with Time-Series Data
01h 04mHow We Build Firebolt
40mFast Access To Your Complex Data #bbuzz
48mHyper Speed! When Big Data Blooms