The Open Source Technology Behind a Tweet

on YouTube ↗22m

" summary

Twitter runs on a JVM stack with GitHub-hosted projects and 80+ open repos, uses MySQL (public fork) behind a sharding layer Gizzard, and stores relationships in FlockDB; unique IDs come from Snowflake and async RPC is handled by Finagle (built on Netty). Real numbers: ~2.8 billion tweets weekly (~5,000 tps steady), peaks of 12,000–25,000 tps and the site stays up. Infrastructure includes a firehose, Earlybird search (Lucene), Hadoop analytics, Apache Mesos with cgroups, and CI via Jenkins.

AI-written summary · about

" chapters (7)

1. 0:00Introduction to Chris Anazik
2. 1:28Twitter's real-time mission
3. 5:19Open source philosophy
4. 7:29Key open source projects
5. 9:32Anatomy of a tweet
6. 15:41Infrastructure and scaling
7. 19:38Conclusion and future plans

" related talks

• 52mPHP at the Firehose ScaleStuart Herbert · PHP UK Conference 2014
• 27mFixing Twitter: Improving the PerJohn Adams · Velocity 2009
• 42mScaling to Billions of Real-time Recommendations DailyAjeet Grewal · Scale By the Bay 2017
• 28mThe Evolution of Twitter's EdgeRyland Degnan · EnvoyCon North America 2021
• 28mOpen Source at FacebookJames Pearce · Collaboration Summit 2015
• 18mHigh-Load Systems: Challenges in Social Network DevelopmentAlexandr Kolobov · Conf42 Platform Engineering 2024
• 22mRebuilding Twitter’s public APISteve Cosenza · Scale By the Bay 2020
• 32m(Probably) a retrospective for SecateurTom Eastman · PyCon AU 2023