New Trends In Storing Large Data Silos With Python
summary
Overview of concrete claims and results: bcolz ctable is a chunked, columnar container and carray handles homogeneous multidimensional data, enabling appends without full copies and chunk-size tuning per storage layer. Blosc is the compressor behind it and blosc decompresses up to 5× faster than memcpy, so transferring compressed blocks into cache then decompressing can be faster than moving raw data. Examples include use on SSD/PCIe, and deployments in OpenVDB, Houdini and Quantopian.
AI-written summary · about
chapters (16)
- 0:00Start
- 1:40Overview
- 2:27Don't Forget Python's Real Strengths
- 3:50The Need For Speed
- 4:23The Daily Python Working Scenario
- 6:26Memory Access Time vs CPU Cycle Time
- 7:37Computer Architecture Evolution
- 8:16Reference Time vs Transmission Time
- 9:25Not All Storage Layers Are Created Equal
- 10:27Trends On Storage
- 11:45Trends On CPUs
- 13:28Why Chunking?
- 14:12Why Columnar?
- 15:51Why Compression (II)? Less data needs to be transmitted to the CPU
- 16:30Accelerating I/O With Blosc
- 17:37Quantopian's Use Case
related talks
43mNew Computer Trends and How This Affects Us
22mBlosc2: Fast And Flexible Handling Of N-Dimensional and Sparse Datasets
43mImprove the efficiency of your Big Data..
01h 27mMastering Large NDArray Handling with Blosc2 and Caterva2
01h 38mUsing Containers for Big Data
26mColumnar Storage: Redefining Data Management for the Modern Era
33mBlaze: crunching data in different formats through a single API.
42mColumnar Storage: Redefining Data Management for the Modern Era