Columnar Storage: Redefining Data Management for the Modern Era
summary
Columnar databases claim big speed and storage gains for analytics by storing columns instead of rows and using columnar file formats and in‑memory layouts like Apache Parquet and Apache Arrow. Examples: ClickHouse reads ~2 billion rows/sec for Disney Plus logs; Cloudflare reports 8 million ingested rows/sec and 7× query throughput improvement; an observability customer shifts from Elasticsearch to save petabytes/day via higher compression. Tools shown include Pandas, DuckDB, Apache DataFusion and a ClickHouse client path that yields ~87× faster DataFrame queries.
AI-written summary · about
chapters (11)
- 0:00Columnar storage introduction
- 1:06Row vs column databases
- 4:16Columnar features and benefits
- 7:07Popular columnar databases
- 9:13Apache Arrow and Parquet
- 12:43Arrow for efficient transfer
- 15:52OLAP databases and pandas
- 17:58Telegraf for metrics
- 19:02Customer use cases
- 23:15Resources and workshop
- 23:57Q&A on row vs column
related talks
42mColumnar Storage: Redefining Data Management for the Modern Era
45mClickHouse: what is behind the fastest columnar database
28mColumnstore Revolution
42mClickHouse: what is behind the fastest columnar database
34mClickHouse® - What is Behind the Fastest Columnar Database?
42mColumnar Storage + Python: Powering Modern Data Science and Analytics
01h 00mColumnar DBMS: The Magic of Data Compression and Super-Fast Queries
37mBenefits of columnar data layout