Managing Capacity

on YouTube ↗20m

" summary

LinkedIn describes a capacity system serving 500 million members across 95,000 servers and 29 MW, with Hadoop at 170 PB, Espresso holding 4.2 PB across 237 databases, and 2 trillion events flowing through Kafka. Capacity methods use metrics every 5 minutes, 98th/97th-percentile quarterly snapshots, buffers, and standard hardware sizing. Built tools include Kafka capacity tool (fabric → topic partition visibility), Dr. Elephant for job tuning, and ReWinder/Sizes for historical and HDFS growth tracking.

AI-written summary · about

" chapters (5)

1. 0:00Introduction
2. 1:45LinkedIn in 2010
3. 8:29Capacity Planning
4. 12:16Capacity Management
5. 15:30Capacity Planning Hadoop

" related talks

• 01h 00mPutting Apache Kafka to UseJay Kreps · Philly ETE 2015
• 29mCapacity Planning with the Universal Scalability LawDaniel Mitterdorfer · Open Source Monitoring Conference 2025
• 50mApache Kafka + Zookeeper = 3.5 Million Writes Per Second by DesignRanganathan "Ranga" Balashanmugam · Great Indian Developer Summit 2015
• 05mPREVIEW: 1 Day, 1 Data Hub, 100 Billion MessagesYuto Kawamura · Kafka Summit San Francisco 2017
• 38mData Pipeline with Apache KafkaMoritz Siuts, Robert von Massow · code.talks 2015
• 32mScaling your Kafka streaming pipeline....Opher Dubrovsky, Ido Nadler · Scale By the Bay 2021
• 32mShould you read Kafka as a stream or in batch? Should you even care?Opher Dubrovsky, Ido Nadler · Berlin Buzzwords 2021
• 38mKafka Message Broker Under High LoadArtem Vybornov · РИТ++ 2017