Migrating Panacea.AI's 5TB/day log platform from Manticore to ClickHouse ... and the lessons learnt
summary
Panacea.AI moves 5 TB of logs per day (154 billion rows, 870k+ inserts/s) from Manticore Search to ClickHouse and cuts projected storage from hundreds of TB to ~20 TB by schema and columnar compression changes. Schema normalization plus per-column codecs and ZSTD + Delta encoding yield multi‑dozen× compression (uploader cites 37×, internal tests ~27×). Most queries are filters (≈85%), so bloom‑filter skip indexes replace heavy inverted indices, trading ~6× slower fuzzy searches (50 ms → ~300 ms) for hardware cost reduction ($300k → $75k) and low‑ops retention via partitioning and TTL.
AI-written summary · about
chapters (7)
- 0:00Log platform migration intro
- 2:50Database constraints for 5TB/day
- 6:21Elasticsearch alternatives
- 8:07Why ClickHouse
- 10:36Fuzzy search limits
- 13:05Validating with peers
- 14:08Migration lessons
related talks
45mClickHouse: what is behind the fastest columnar database
42mClickHouse: what is behind the fastest columnar database
28mColumnstore Revolution
34mClickHouse® - What is Behind the Fastest Columnar Database?
26mHandling Large Volumes of Immutable Structured Data With ClickHouse
17mBuilding Real-Time Data Lake with ClickHouse
18mSwitching Jaeger Distributed Tracing to ClickHouse® to Enable APM
17mUsing ClickHouse® Database to Power an Analytics and Customer Engagement Platform