How to Build a Data Streaming Platform on Kafka

on YouTube ↗46m

" summary

Covers building a Kafka-based data streaming platform: how to ingest data into Kafka from any source, handle schema evolution, and other developer pitfalls. Describes scaling and opening the platform to many teams (Raiffeisen Bank runs it for 20 teams and 200+ topics) and shows how to transform, join, and aggregate Kafka data using SQL (and Java when needed).

AI-written summary · about

" chapters (20)

1. 0:00Introduction
2. 1:46What is a streaming platform
3. 5:17Kafka's role
4. 6:43Loading data with Kafka Connect
5. 8:49Data validation and schema
6. 9:30Real-time data processing
7. 13:25Exporting to analytics
8. 15:09Federated data governance
9. 16:56Low entry threshold
10. 18:20Common issues and fixes
11. 20:07Security and access control
12. 22:37Resources and quotas
13. 23:44Deployment challenges
14. 25:50Strimzi for Kafka on Kubernetes
15. 26:33Data ingestion transparency
16. 28:17Flink data transformations
17. 31:07Real-time data visualization
18. 32:54Time Series DB selection
19. 34:39Platform roadmap
20. 36:01Q&A