Scaling Pinterest
summary
Startup grows from a handful to 150+ engineers and hundreds of millions of users by simplifying after a chaotic phase of dozens of datastores; early experimentation with Cassandra, MongoDB, mBase and others fails at 3 a.m., so the stack converges to MySQL + memcached + Redis core. A dedicated data pipeline is built: Kafka -> S3 logging pipeline feeding EMR/Redshift and stream consumers; follower graphs live in Redis, home feeds move to HBase, service connections use ZooKeeper, and deployment speed jumps to deploys every 30 seconds despite doubling every 1.5 months.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:50Evolution of Pinterest
- 11:10Technologies
- 18:02Sharding
- 23:35Amazon
- 24:09Dynamo
- 24:52ELB
- 25:09Python
- 26:19Java
- 27:16Memcache
- 28:30Core Data
- 28:58Redis
- 30:59Pinterest follower graph
- 32:26Why HBase
- 34:25Rich Pin Details
- 34:46Technology Failure
- 37:12Second Chance
- 38:51Other Things
- 42:19Be Testing
- 43:06Looking Forward
- 43:42Have Fun
related talks
44mEngineering for Engineering's Sake
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
36mBuilding a Self-Service Streaming Platform at Pinterest
41mEngineering for Engineering's Sake
30mHow We Engineered Value in Times of COVID?
52mPHP at the Firehose Scale
47mScaling: from 0 to 25 million users
18mHigh-Load Systems: Challenges in Social Network Development