Polyglot, Fault-Tolerant Event-Driven Programming with Kafka....

on YouTube ↗35m

" summary

Wix describes a Kafka-based event infra called Greyhound that handles 1.5 billion business messages per day across 1,500 microservices and guarantees eventual delivery via retry topics, local persistent queues and a resilient producer that writes to disk then flushes to Kafka. A node-level scavenger DaemonSet recovers on-node unsent messages after pod rescheduling. Concurrency uses Z fiber-style threads so a single consumer can handle 80 partitions → 800 msg/s instead of 80 JVM threads, and sidecar vs DaemonSet deployments trade memory for isolation.

AI-written summary · about

" chapters (11)

1. 0:00Introduction and welcome
2. 1:28Talk overview and Wix context
3. 4:19Event-driven vs request-reply systems
4. 6:09Kafka as message broker solution
5. 8:21Greyhound library introduction
6. 9:48High performance with Greyhound
7. 14:06Fault tolerance strategies
8. 18:26Kubernetes deployment challenges
9. 22:04Polyglot environment with gRPC
10. 27:11Sidecar vs daemon set proxies
11. 32:28Summary and Greyhound open source

" related talks

• 30m8 Lessons Learned from using Kafka with 1000 microservicesNatan Silnitsky · Scale By the Bay 2020
• 52m10 LESSONS LEARNED FROM USING KAFKA IN 1000 SCALA MICROSERVICESNatan Silnitsky · Scalar 2020
• 54mLessons learned from working with 2000 event-driven microservicesNatan Silnitsky · GeeCON 2023
• 37mKafka Based Global Data Mesh at WixNatan Silnitsky · DevDays Europe 2022
• 46mLessons Learned From Working with 2000 Event-Driven MicroservicesNatan Silnitsky · Build Stuff 2022
• 29mGreyhound - Powerful Pure Functional Kafka libraryNatan Silnitsky · Scala Love in the City 2021
• 43mGreyhound - Powerful Pure Functional Kafka libraryNatan Silnitsky · LambdaConf 2020 Global Edition
• 41mBattle-tested event-driven patterns for your microservicesNatan Silnitsky · Scale By the Bay 2021