Greyhound - Powerful Pure Functional Kafka library

on YouTube ↗29m

" summary

Greyhound is an open source Scala Kafka SDK rewritten with ZIO fibers for per-partition concurrency and used across ~1500 Wix microservices. It provides automatic per-partition dispatching, configurable nonblocking and blocking retry policies implemented as retry topics per interval (inspired by Uber), a resilient disk-backed producer with Kubernetes handoff, header-based request context injection into handlers, defaults to simplify setup, and metrics/dashboards. Current release is 0.1.5 and the rewrite reduces boilerplate while improving throughput and fault tolerance.

AI-written summary · about

" chapters (13)

1. 0:00Introduction to Greyhound
2. 1:55Kafka fundamentals overview
3. 4:03Removing boilerplate with Greyhound
4. 7:17Functional design benefits
5. 9:22Greyhound consumer internals
6. 12:14Parallel consumption with fibers
7. 14:42Error handling and retries
8. 19:20Resilient producer for failures
9. 21:08Context propagation in Kafka
10. 24:19Pausing and monitoring features
11. 25:44Greyhound use cases at Wix
12. 27:13Rewriting Greyhound in ZIO
13. 28:19Open source and resources

" related talks

• 43mGreyhound - Powerful Pure Functional Kafka libraryNatan Silnitsky · LambdaConf 2020 Global Edition
• 35mPolyglot, Fault-Tolerant Event-Driven Programming with Kafka....Natan Silnitsky · Devoxx Ukraine 2020
• 52m10 LESSONS LEARNED FROM USING KAFKA IN 1000 SCALA MICROSERVICESNatan Silnitsky · Scalar 2020
• 30m8 Lessons Learned from using Kafka with 1000 microservicesNatan Silnitsky · Scale By the Bay 2020
• 37mKafka Based Global Data Mesh at WixNatan Silnitsky · DevDays Europe 2022
• 54mLessons learned from working with 2000 event-driven microservicesNatan Silnitsky · GeeCON 2023
• 46mLessons Learned From Working with 2000 Event-Driven MicroservicesNatan Silnitsky · Build Stuff 2022
• 41mReducing Microservice Complexity with Kafka and Reactive StreamsJim Riecken · Scala Days Berlin 2016