Flink powers Houzz near-realtime pipeline & Kubernetes logging observability
summary
Within a year Apache Flink becomes Houzz’s near‑real‑time engine, running 15+ production jobs on Kubernetes and processing 10+ TB/day into Kafka topics; an event hub performs multi‑stage enrichment (joins to session/login), achieves >98% enrichment with low 95th‑percentile latency, and uses RocksDB on SSD with background compaction for deduplication. Downstream stores include Redis, Elasticsearch and Cassandra. Flink also replaces Logstash for Kubernetes logging, enabling joined logs, JS processors, and an Istio cross‑AZ tracing view that speeds debugging after a 100% k8s traffic migration.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 0:45What is House
- 1:10Agenda
- 1:43Flink stats
- 2:24Data pipeline overview
- 4:44Multistage event data enrichment
- 7:06Nearrealtime pipeline
- 7:43Flink requirements
- 11:11Kubernetes logging observability
- 11:57Requirements
- 13:15Architecture
- 14:48Case study
- 15:24Flink approach
- 16:02Flink visualization
- 17:28Flink use cases
- 19:34Future work
related talks
26mKubernetes & real-time analytics: How to connect these worlds with Apache Flink
42mStream Processing with Apache Flink
43mApache Flink in Production with Ververica Platform
44mLets Deeploy Flink - Uncovering Hidden Depths of Yarn, Docker, Kubernetes & Beyond
30mCloudflow: Spark, Flink, and Akka Working together on Kubernetes
24mKeynote: The Easiest Way to Get Operational With Apache Flink
25mFlink on Karmada: Building Resilient Data Pipelines on Multi-Cluster K8s
37mSpark and Flink Running Scalable in Kubernetes #bbuzz