InStream: Large Scale Distribution...
summary
Describes a Python service built on SaltStack and Kafka that uses a BitTorrent-like P2P distribution model to deploy services asynchronously across LinkedIn’s datacenters. Covers how the system handles scale by avoiding centralized bottlenecks and providing features such as rate limiting and fast rollbacks.
AI-written summary · about
chapters (10)
- 0:00Start
- 3:47Generic Deployment System
- 6:40Dist-tree
- 8:06Bittorent protocol
- 9:21Instream Architecture
- 11:29Libtorrent as torrent client
- 12:20Saltstack as remote execution engine
- 14:36API and CLI
- 17:06Challenges and Learnings
- 20:26Future Goals
related talks
37mLet's build our own message streaming platform
36mStreamlio: Towards a Unified End-to-end Solution
28mBuild your next-generation stream platform based on Apache Pulsar
22mJoining Dozens of Data Streams in Distributed Stream Processing Systems
30mEvent Streaming and the Future of Applications
24mPushing Your Streaming Platform to the Limit
03mSijie Guo Interview
55mDemystifying Stream Processing with Apache Kafka