Building an Analytics Platform
summary
Simple builds a Redshift-based warehouse, moving from ad‑hoc queries to near‑real‑time dashboards by batching JSON into S3 via a RabbitMQ message queue, then loading into Redshift on a 32‑node cluster (32 nodes, ~3 TB). Four engineers spend about a year to build an ingestion pipeline with a batcher, loader and per‑source ingesters, write Python admin tools like shift-manager and Celery schedulers, use Scala for production services, and later migrate messaging to Kafka; the system works but is heavy to maintain.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 1:07What is Analytics
- 2:37About Simple
- 4:17The Portland Timbers
- 4:45Simple
- 5:37Safe to Spend
- 6:30User Interactions
- 10:19Data Warehouse
- 11:30Amazon Redshift
- 12:59Data Pipelines
- 13:22Visual Representation
- 17:40Load Data into S3
- 17:59Redshift Interface
- 19:44Message Queue
- 21:52Message Format
- 22:56ingestion layer
- 24:49good idea
- 27:11Kinesis Firehose
- 27:46Python
- 28:00No Python
- 29:51Shift Manager
- 31:19Sequel Alchemy
- 32:55Deep Copy
- 34:34Celery
- 35:01What is Celery
- 35:59Scheduled Tasks
- 36:59Questions
related talks
43mData warehousing reinvented for today's needs
30mData Engineering building your BI infrastructure from scratch
41m7 steps to build your own data pipeline
30mRaw Data to Business Insights: What You Need to Build a Modern Data Lake
23mOne day with Redshift
17mDemocratising data analytics
01h 02mData warehousing reinvented for today’s needs
25mBuilding an analytics API