Running Apache Flink on AWS
summary
Compares three concrete ways to run Apache Flink on AWS: Amazon EMR for managed Hadoop-style clusters, Amazon EKS for full-control Kubernetes deployments, and Amazon Kinesis Data Analytics as a stateful serverless Flink service that runs on EKS under the covers. Kinesis Data Analytics manages Flink lifecycle, uses RocksDB and S3 for state, exposes capacity as Kinesis Processing Unit (KPU) (1 vCPU, 4 GB), auto-scales CPU-bound parallel jobs, stores up to 1,000 savepoints but currently cannot import a savepoint into a different application.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:28Streaming Data Services
- 5:34Advantages and Disadvantages
- 8:48How to run Apache Flink
- 9:45What does Kinesis manage
- 11:14Application lifecycle management
- 16:21Checkpoints
- 17:37Documentation
related talks
44mBuild a Real-time Stream Processing Pipeline with Apache Flink on AWS
40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobs
38mApache Flink: A Very Quick Guide
42mManaging Apache Flink on Kubernetes - FlinkK8sOperator
41mBeyond the Hype: When Apache Flink Solves Real Problems
45mApache Flink — Real-Time Data Processing
20mGetting Started with Event Stream Processing via Apache Flink on Microsoft Azure
42mApache Flink deep-dive #bbuzz