8 Lessons Learned from using Kafka with 1000 microservices

on YouTube ↗30m

" summary

Wix runs ~1,500 microservices and 1.5 billion Kafka messages per day and builds a shared Kafka layer called Greyhound rewritten on ZIO. Greyhound adds parallel consumption via fibers, automatic commit and header-based context propagation, built-in retry topics plus a blocking-retry policy and per-service dashboards. Self-service back-office and schema discovery reduce ops load, Prometheus/Grafana metrics trigger alerts on consumer lag, Node.js uses a gRPC sidecar to reuse Java/Scala features, and clusters are proactively trimmed or migrated to Confluent Cloud.

AI-written summary · about

" chapters (4)

1. 0:00Start
2. 9:06BLOCKING POLICY HANDLER
3. 10:48CONTEXT PROPAGATION
4. 18:44METRICS PUBLISHING

" related talks

• 52m10 LESSONS LEARNED FROM USING KAFKA IN 1000 SCALA MICROSERVICESNatan Silnitsky · Scalar 2020
• 46mLessons Learned From Working with 2000 Event-Driven MicroservicesNatan Silnitsky · Build Stuff 2022
• 54mLessons learned from working with 2000 event-driven microservicesNatan Silnitsky · GeeCON 2023
• 35mPolyglot, Fault-Tolerant Event-Driven Programming with Kafka....Natan Silnitsky · Devoxx Ukraine 2020
• 37mKafka Based Global Data Mesh at WixNatan Silnitsky · DevDays Europe 2022
• 43mBattle-tested event-driven patterns for your Microservices archit.Natan Silnitsky · Devoxx Ukraine 2020
• 41mBattle-tested event-driven patterns for your microservicesNatan Silnitsky · Scale By the Bay 2021
• 29mGreyhound - Powerful Pure Functional Kafka libraryNatan Silnitsky · Scala Love in the City 2021