Getting Started with Flink for Small Teams w/ RabbitMQ and Kubernetes

on YouTube ↗17m

" summary

Walks through a concrete small-company pipeline: Twitter ingestion into a Flink job that buckets hourly analytics and bulk indexes to Elasticsearch, then explains why a growing monolithic job causes restarts, checkpointing and scaling pain and how inserting RabbitMQ as message queues decouples pieces. Uses Kubernetes on EKS with Ververica for deployment, started Flink in production after ~1 year of usage and RabbitMQ for ~4 years, reports one month to set up the cluster, now running 6 Flink jobs processing 4 million events/day and planning a future move to Pulsar.

AI-written summary · about

" chapters (6)

1. 0:00Intro
2. 1:13Analytics for Dashboards
3. 5:03Deploying it
4. 9:58Combo Flink Job Sketch
5. 12:35Messaging Queues
6. 15:39Chaining Jobs

" related talks

• 26mKubernetes & real-time analytics: How to connect these worlds with Apache FlinkAlbert Lewandowski · Berlin Buzzwords 2021
• 37mSpark and Flink Running Scalable in Kubernetes #bbuzzFrank Conrad · Berlin Buzzwords 2018
• 24mKeynote: The Easiest Way to Get Operational With Apache FlinkKonstantin Knauf · Flink Forward Virtual 2020
• 43mApache Flink in Production with Ververica PlatformSeth Wiesman · Flink Forward Virtual 2020
• 33mFlink in ContainerlandPatrick Lucas · Flink Forward Berlin 2017
• 44mLets Deeploy Flink - Uncovering Hidden Depths of Yarn, Docker, Kubernetes & BeyondNaci Simsek · Big Data Conference Europe 2025
• 22mBuilding a Kubernetes Operator for Apache Flink in JavaAttila Mészáros, Gyula Fora · Community Over Code EU 2024
• 37mIntegrate Flink with Kubernetes nativelyYang Wang · Flink Forward Virtual 2020