Payment workflow with Ruby and Clickhouse
summary
Payment processing moves from slow serial stored procedures to parallel Sidekiq batches keyed by UUIDs and stored state in Redis, with per-batch idempotency tracked in temporary tables so retries are safe. CSVs stream to S3 (avoiding local files, saving ~3s per chunk) and ClickHouse on EC2 ingests those CSVs via S3 connectors using ruby ODBC, handling 200M weekly rows (15 billion total) and doing 200M inserts in ~15 minutes; reporting returns 2.5 GB to S3 in 9 seconds and end-to-end jobs finish in ~3 hours instead of days.
AI-written summary · about
chapters (9)
- 0:00Introduction and context
- 3:34Problem at scale with old architecture
- 7:47Making workers idempotent
- 11:59Optimizing bulk database transactions
- 15:14Switching to denormalized reporting
- 17:00Choosing ClickHouse for analytics
- 19:27New architecture overview
- 20:08Impact and performance gains
- 21:54Q&A session
related talks
26mHandling Large Volumes of Immutable Structured Data With ClickHouse
34mClickHouse® - What is Behind the Fastest Columnar Database?
55mBuilding Fast ClickHouse Analytics for MySQL Transaction Data Using Kubernetes
35mProcessing data at scale with Rails
42mClickHouse: what is behind the fastest columnar database
22mIt’s all about the db - a journey to accommodate 100k events per second
30mBuilding For Fast
17mUsing ClickHouse® Database to Power an Analytics and Customer Engagement Platform