Magnit Tech: Inventory and Pricing Services in Go — Handling Large Data Streams
summary
Describes building Magnit Tech’s inventory and pricing system in Go to handle very large data volumes (about 30k stores, 10–20k SKUs, ~500 million rows) and very high write load (~150k writes/sec). Covers technical challenges encountered at that scale and the system design and engineering approaches used to manage throughput and data volume.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:11What are inventory and prices
- 3:39Existing system problems
- 5:47Architecture choice
- 7:31Storage and broker choice
- 9:18Tech stack
- 10:01Data loading implementation
- 11:07PostgreSQL tuning
- 13:57Kafka setup
- 16:26Working with Go
- 18:17PostgreSQL interaction
- 20:26Stress testing
- 22:52Load results
- 23:13Monitoring and alerts
- 24:15Business analytics
- 25:38Error policy
- 25:59Summary and conclusions
- 26:44Q&A
related talks
17mExtreme Engineering-Elements of Building a High -scale Broking Application
53mGo at Mainframe Scale
28mManaging and Scaling Real Time Data Pipelines using Go
26mDemocratizing Data at Go-JEK
18mHomegrown Technology at Scale
31mHow Go is Making us Faster
41mReliability of Data Delivery in Stream Processing
24mRetail Transactions at scale for AI - From PoC to production