Turning Off‑the‑Shelf Tools into a Petabyte‑Scale System
summary
A practitioner describes building a petabyte-scale data system using existing (import-substituted) tools for collection, processing, and storage, including how to choose collection and processing approaches when many external data sources are involved. The talk also covers systematization with ready-made storage/import-replacement solutions and argues that security concerns need not be a major obstacle.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:08Architect's journey to servers
- 7:50System views and tool selection
- 13:48Non-functional requirements
- 19:06Insights, failures and optimization
- 24:27Final costs and architecture reuse
- 29:31Bonus: Checklist for tough tasks
- 31:17Q&A
related talks
56mResistance is Futile: What You Need to Know about Big Data
33mAnalytics Platform at Petabyte Scale
20mData in the Apache "big data" ecosystem
49mBuilding a 100‑Petabyte Analytics Warehouse
18mOur Analytical Data Journey – from Paper to Petabytes
24mA Gentle Introduction to Building Data-Intensive Applications
26mPetabyte Scale Datalake Table Management with Ray, Arrow, Parquet, and S3
57mBig Data: Threat or Opportunity?