So You’ve Got Yourself a Kafka: Event-Powered Rails Services
summary
Explains Kafka basics and concrete Rails tooling: ruby-kafka, Racecar, DeliveryBoy and Karafka wrappers; recommends monitoring offsets and using consumer groups tied to topic partitions to parallelize work. Cites Netflix/LinkedIn/Microsoft sending ~a trillion messages/day. Warns ruby-kafka operates with an at-least-once model so consumers must handle duplicates (UPSERT or idempotency), describes simulating outages by rewinding offsets in Gameday tests, and contrasts JSON payloads with AVRO + schema registry and AVRO::Builder for safe schema evolution.
AI-written summary · about
chapters (24)
- 0:00Start
- 2:07DISTRIBUTED STREAMING PLATFORM
- 3:32Producers
- 4:02How is this different from Sidekig or Resque?
- 10:36High Throughput
- 11:17Consumer Agreement is Tricky
- 12:32Scalable and Fault Tolerant
- 13:17KAFKA POWERED SERVICES
- 14:39Challenge 1: Upstream service is responsible for downstream availability
- 15:41Challenge 2: New downstream services require changes to the upstream service
- 16:22New Challenge: Dependencies are no longer explicit
- 16:57The Many Meanings of Event-Driven Architecture
- 19:40Event-Sourced Architecture
- 21:56Command Query Responsibility Segregation
- 22:12CRUD vs CORS
- 23:38When to use it?
- 24:28PRACTICAL CONSIDERATIONS
- 24:50SLOW CONSUMERS
- 27:38ruby.kafka#what-to-monitor
- 29:21Design Consumers for Duplicated Events
- 30:32Downside to Data Flexibility
- 31:30Schema Registries
- 32:15JSON
- 34:00Avro::Builder
related talks
33mI Can’t Believe It’s Not A Queue: Using Kafka with Rails
42mSimplifying Logs, Events and Streams: Kafka + Rails
31mEvent Streaming on Rails
38mEffective Data Synchronization between Rails Microservices
41mKafka as a Platform: The Ecosystem from the Ground Up
39mKafka as a Platform: The Ecosystem from the Ground Up
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
01h 00mKafka as a Platform: the Ecosystem from the Ground Up