Apache Kafka in 1 hour for C# Developers

on YouTube ↗53m

" summary

Step‑by‑step account of a production event system that connects Patreon webhooks to a C# stack: Azure Functions receive webhooks, messages go to Confluent Cloud topics, code uses KafkaFlow on .NET to produce and consume with worker threads and middleware, keys = emails to guarantee partition order, and a compacted topic stores snapshots. Real problems covered: GitHub API throttling, duplicated emails caused by at‑least‑once delivery and solved with idempotency checks, and handling replay by reading topics from the beginning; this setup runs in production for a channel with 5.6K subscribers.

AI-written summary · about

" chapters (17)

1. 0:00Introduction to Kafka
2. 3:52Project context and goals
3. 7:07Setting up the infrastructure
4. 11:02Core Kafka concepts
5. 13:01Partitions and brokers
6. 15:56Implementing the producer
7. 19:26Producer best practices
8. 21:58Message structure and keys
9. 25:00Ordering and partitioning
10. 27:19Implementing the consumer
11. 30:13Consumer configuration
12. 33:23Acknowledgements and tradeoffs
13. 37:23Real-world event handling
14. 39:42Handling third-party APIs
15. 42:09Consumer groups explained
16. 45:20Evolutionary architecture
17. 48:00Compacted topics and recap

" related talks

• 49mApache Kafka in 1 hour for C# DevelopersGui Ferreira · NDC Porto 2023
• 55mApache Kafka in 1 hour for C# DevelopersGuilherme Ferreira · NDC London 2023
• 01h 03mApache Kafka Event streaming platform for .NET developersViktor Gamov · NDC Sydney 2019
• 01h 02mKafka for .NET DevelopersIan Cooper · NDC Sydney 2024
• 57mUsing Kafka for Real-Time Data Ingestion with .NETKevin Feasel · NDC Oslo 2018
• 01h 00mKafka for .NET DevelopersIan Cooper · NDC Oslo 2024
• 01h 05mKafka for .NET DevelopersIan Cooper · NDC London 2024
• 29mApache Kafka and the Next 700 Stream Processing SystemsJay Kreps · Strange Loop 2015