To Petascale and Beyond: Apache Flink in the Clouds
summary
Shows how Apache Flink plus Gelly generates R-MAT and worst-case graphs at petabyte scale (trillions to quadrillions of triplets) and implements Gelly graph generators for algorithms like triangle counting, clustering coefficient, common neighbors, Jaccard and Adamic–Adar. Describes engineering: implicit operator reuse, object reuse that yields ~5% gains, custom serializers, off‑heap memory, memoryFraction≈0.9, multi‑disk temp dirs, tuning EBS IO and Netty buffers, and avoiding Java SSL CPU by using the Python CLI. Uses Java Flight Recorder for profiling, publishes a benchmark harness and scripts on GitHub, and reports a 100 TB CloudSort run (costs ≈ half prior champion; lost due to S3 throttling).
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:15My Project Background
- 2:57What is Gelly?
- 5:52Glossary
- 7:25Clustering Coefficient
- 8:31Enumerating Triangles
- 9:16Implicit Object Reuse
- 10:42Implicit Operator Reuse
- 12:49GraphAnalytic
- 14:07Similarity Measures
- 14:53Common Neighbors
- 15:33GroupedCross
- 18:16Flink Serialization
- 23:08Amazon Web Services
- 24:53Memory Allocation
- 26:41Configuring Network Buffers
- 27:21Python statsd server
- 30:30Java SSL Performance
- 31:30Netty Configuration
- 34:07Macro Benchmarks
- 34:46Custom Types
- 38:41Java Flight Recorder
related talks
42mApache Flink deep-dive #bbuzz
45mApache Flink — Real-Time Data Processing
38mApache Flink: A Very Quick Guide
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
36mSources, Sinks, and Operators: A Performance Deep Dive
25mData: Moving with the flow
47mBring Flink to Large Scale Production
36mApache Flink in the Cloud-native Era