Big and Small Data at @Twitter
summary
Twitter handles about 155 million tweets per day (roughly 1,700/sec average, peaks near 7,000/sec) and adds ~460,000 accounts daily. Core storage uses MySQL nodes (typical DB server: HP DL380, 72 GB RAM, 24-disk RAID10) for datasets up to ~1.5 TB, with a custom layer: gizzard for sharding, flockdb for edges, t-bird as the canonical tweet store and snowflake for distributed ID generation. Hadoop/HDFS and Vertica run batch analytics, Cassandra sees high-write use cases, and the new stack reduces costly temporal sharding that filled clusters every three weeks.
AI-written summary · about
chapters (7)
related talks
30mThe Distributed Database Behind Twitter
35mStorage, Then and Now
28mAn Architecture to Tweet Them All
05mBret Piatt interviewed at Velocity 2010
45mRealtime Search at Twitter
41mInside Twitter's Realtime Delivery of Tweets for Enterprise Customers
48mHyper Speed! When Big Data Blooms
22mIt’s all about the db - a journey to accommodate 100k events per second