Multi-tenant Flink-as-a-Service on YARN
summary
Flink runs on YARN inside HopsWorks as a UI-driven, open-source service deployed since June 2016 at www.hops.site, offered on a 54‑machine cluster (~10 TB RAM, ~100 users) with project-based isolation: each project gets an HDFS subtree, quotas and metered CPU hours that decrement on every container allocation. Kafka topics are project-scoped and protected by per-project TLS certificates injected as YARN local resources, and a provided Kafka util one-line API hides cert and schema plumbing. Name‑node metadata scales linearly to ~30 nodes and the filesystem metadata moves into NDB to reach terabyte heaps.
AI-written summary · about
chapters (7)
- 0:00Big data and Hops overview
- 3:24Architecting for scalability
- 11:00Project-based multi-tenancy
- 15:34Security and authentication
- 19:16Simplifying Flink operations
- 24:52Demo of the Hops platform
- 39:46Summary and Q&A
related talks
45mApache Flink — Real-Time Data Processing
21mFlink's application at Didi
42mApache Flink deep-dive #bbuzz
32mFlink architecture and automation in Intesa Sanpaolo
46mFlink SQL in 2020: Time to show off!
31mFlink @WORKDAY: scaling of ingestion data pipelines to S3
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
40mLarge-scale near-real-time (NRT) data analytics platform empowered by Apache Flink