Our experience at Etsy com with building “Big Data” analytic capabilitie
summary
Etsy moves analytics off a single Postgres master and slow Hadoop batches into a columnar cluster using Vertica, cutting typical aggregation runtimes by orders of magnitude (speaker claims ~100x). To populate and keep Vertica fresh two open tools are built: hleb for fast bulk COPY-style snapshots (10M rows in seconds) and autoshlep for continuous replication including deletes (typical sync under 15 minutes). Scale context: ~20M users, ~1M sellers, ~1B listings, 200 engineers; the pipeline reliably serves interactive BI.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 3:58Etsy’s early architecture and challenges
- 12:46Transition to sharded architecture
- 21:26Business intelligence needs and PostgreSQL replication
- 27:26Introducing Vertica for analytics
- 33:16Building data replication tools
- 39:27Live demo of data sync and queries
- 45:01Q&A and closing remarks
related talks
30mAn architecture for big data: Supporting journalism through transparency in advertising
43mArchitecting for Speed & Scale
01h 35mDeep Dive into the Big Data Landscape
41mBig data as we know it @bol.com
53mBig Data: from mammoth to elephant
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
49mBuilding a 100‑Petabyte Analytics Warehouse
47mRethinking Relational Database CRUD Design Patterns Since We Live in a Big Data World