A Deep Dive into Apache Kafka This is Event Streaming

on YouTube ↗02h 05m

" summary

Breaks down Kafka internals (brokers, leaders/followers, replication and min.insync.replicas behavior that blocks producers when replicas lag), shows zookeeper metadata and the KIP500 migration, details producer settings (acks=all, retries, idempotence, batching/compression) and consumer groups/offset commit modes, demos Kafka Connect (source/sink, REST API, connect distributed mode, 80+ connectors, CDC use cases), and demonstrates Kafka Streams’ DSL with KStream/KTable (word-count example that creates repartition and changelog topics). Covers running Kafka on Kubernetes and operational monitoring/alerts.

AI-written summary · about

" chapters (57)

1. 0:00Introduction
2. 1:27What is Kafka
3. 6:45Getting started with Kafka
4. 9:27Kafka Cluster
5. 16:11Zookeeper
6. 19:56Zookeeper History
7. 24:42Topics
8. 25:38Kafka Producer
9. 26:04Kafka Producer Properties
10. 26:16Kafka Producer Send
11. 26:43Kafka Library
12. 27:17Spring Boot Starter
13. 27:31Producers
14. 29:00Retention Period
15. 29:48Compaction
16. 30:44Partitions
17. 32:01Commitment
18. 32:29Middleground
19. 33:21Retry
20. 34:02Idempotency
21. 34:33Batch Records
22. 34:55Compression
23. 35:36Consumer
24. 36:36Consumer Groups
25. 38:45Consumer Offsets
26. 39:47Consumer Config
27. 39:57Why Consumer Groups
28. 40:51Committing offsets
29. 41:09Automatic Committing
30. 43:03Partition Scale
31. 43:55Synchronous
32. 44:40Kafka Monitoring
33. 45:15Kafka Metrics
34. 46:13Kafka Security
35. 46:58Kafka Internal SSL
36. 47:11Kafka Broker SSL
37. 48:03Kafka External SSL
38. 48:27Security
39. 48:42Authorization
40. 49:33Debugging
41. 50:52Kafka Log Forge
42. 51:08Debugging Commands
43. 51:28Check Consumer Groups
44. 51:45Check Producers
45. 51:59Minimum in sync replicas
46. 53:37Metrics
47. 53:57Exception
48. 54:16Ensure Leaders are Preferred
49. 54:37Preferred Leadership Election
50. 54:48Partition Distribution
51. 55:11Partition Consumers
52. 55:27Four Letter Words
53. 56:15Zookeeper ZebCase Honey
54. 56:43What if you want to contribute
55. 56:58And this is essentially when
56. 57:37pull request in Kafka
57. 58:09start again

" related talks

• 01h 00mKafka as a Platform: the Ecosystem from the Ground UpRobin Moffatt · NDC Oslo 2020
• 39mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia Europe 2020
• 01h 00mKafka as a Platform: the Ecosystem from the Ground UpRobin Moffatt · NDC Oslo 2020
• 27mRunning Kafka in Kubernetes a practical guideKatherine Stanley · Voxxed Days Microservices 2018
• 41mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia November 2020
• 01h 03mApache Kafka Event streaming platform for .NET developersViktor Gamov · NDC Sydney 2019
• 31mEvent Streaming with Kafka StreamsViktor Gamov · Scale By the Bay 2020
• 45mManage, secure, and socialize Kafka topicsMichel Lara · apidays Paris 2021