Tuning Flink Filesystem Plugins for State Persistence
summary
Flink jobs with hundreds of stateful operators generate many small files that trigger GCS/S3 API rate limits and huge JVM churn because the Hadoop GCS connector allocates a 64 MB upload chunk buffer per file. The fix is to increase state backend thresholds so task managers batch state to the job manager, cutting checkpoint duration from ~15–20s to ~1–2s, reducing API calls, CPU and heap pressure and OOMs. For S3 set fs.s3a.endpoint NAT gateway and use a local buffer dir /tmp.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:35Quick Bio: Scott Kidder
- 2:04Flink Apps: Mux Data Alerting
- 2:39Flink Apps: Mux Video Log Enrichment
- 3:22Flink Apps: Mux Video Media Usage Proce
- 4:02Comparison of Applications
- 4:47What is state?
- 5:18Checkpointing
- 5:50State Backends
- 6:33Troubleshooting HDFS
- 7:19HDFS Underreplicated Blocks
- 8:25HDFS Alerting Issues
- 9:03Path to Google Cloud Storage
- 9:37Configuring GCS Connector
- 10:37GCS, the struggle continues...
- 11:38GCS Solved!
- 12:57Updates to State Backend Thresholds
- 14:29Why did CPU usage drop?
- 17:36HDFS core-site.xml for S3
- 18:07Conclusion
related talks
32mGetting into Low-Latency Gears with Apache Flink
29mWhen your stream doesn't stream very well
29mA brief history of time with Apache Flink
23mBuilding a network stack for optimal throughput / low-latency trade-offs
19mCome Hell or Still Water: Alleviating Tail Latency in Cloud Block Store
27mScaling Log Collection with Fluent Bit
20mFrom Bytes to Brilliance : Optimizing Large File Delivery in REST APIs
44mOptimizing storage price and performance with Amazon S3