Diving into the World of Apache Kafka @ Michelin: The Balance Between Dev and Ops
summary
A Michelin engineering team presents how they operate a large Apache Kafka estate (Confluent Cloud plus 60 on‑prem clusters, ~4,500 topics/15,000 partitions) and two open‑source projects they use to balance developer and ops needs: Ns4Kafka, which provides namespace isolation, autonomous resource management, desired‑state configuration validation and a CLI (kafkactl) for ‘infrastructure as code’; and Kstreamplify, a Kafka Streams framework that simplifies app initialization, native error handling, common utilities (e.g., deduplication), optional Kubernetes integration and a Spring Boot layer.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:04Kafka ecosystem at Michelin
- 5:19Business and technical context
- 8:27Castimpify for Kafka Streams
- 10:34Castimpify demo with use case
- 17:33Error handling and poison messages
- 24:38NS4Kafka for operability
- 27:25NS4Kafka benefits
- 32:01NS4Kafka deployment demo
- 41:16Castimpify and NS4Kafka integration
- 43:46Conclusion
related talks
40mKafka, DevOps...And Resilience for All
33mMicroservices, Events, and Breaking the Data Monolith with Kafka
41mKafka as a Platform: the Ecosystem from the Ground Up
50mIntroduction to Apache Kafka
39m7 Years in Kafka Land
30mArchitecting Applications With Multiple Open-Source Big Data Technologies
01h 00mPutting Apache Kafka to Use
02h 48mKafka: Map of Pitfalls for Developers and Ops