Using Kafka for Real-Time Data Ingestion with .NET

on YouTube ↗57m

" summary

Builds a .NET pipeline using the Confluent .NET client and F# that produces, enriches and consumes flight data, reading CSV, caching airport lookups from SQL Server, serializing enriched records as JSON and writing aggregates back via a TVP stored procedure. Shows real numbers: 3.3M records in ~3 minutes from a laptop (≈60k rec/s observed, up to ~1M rec/s on proper servers). Uses topic partitions and consumer groups with stored offsets to scale and avoid duplicate work.

AI-written summary · about

" chapters (16)

1. 0:00Start
2. 0:20Curated Sequel
3. 1:29Why Kafka Is Interesting
4. 7:46The Kafka Model
5. 10:46Partitions
6. 12:09Consumers
7. 12:48Consumer Group
8. 19:53Confluence Dotnet Library for Kafka
9. 22:01Create a Consumer Group
10. 23:01Key Based Partitioning
11. 25:53Nuget Packages
12. 26:45Enrichment
13. 29:10Sequel Connection
14. 30:58Building a Dictionary
15. 31:22Configuration Settings
16. 33:18Compaction of Messages

" related talks

• 01h 03mApache Kafka Event streaming platform for .NET developersViktor Gamov · NDC Sydney 2019
• 49mApache Kafka in 1 hour for C# DevelopersGui Ferreira · NDC Porto 2023
• 53mApache Kafka in 1 hour for C# DevelopersGuilherme Ferreira · Copenhagen Developers Festival 2023
• 01h 02mKafka for .NET DevelopersIan Cooper · NDC Sydney 2024
• 01h 05mKafka for .NET DevelopersIan Cooper · NDC London 2024
• 01h 00mKafka for .NET DevelopersIan Cooper · NDC Oslo 2024
• 14mPower real-time data streams from anywhere to Cosmos DB with ConfluentFlorence Sullivan · Microsoft Build 2023
• 25mBeyond Kafka: Leveraging Event-Driven Architectures with Go and NATS JetStreamLudovico Russo · GoLab 2024