Apache Kafka as a Foundation for Building Bicycles

on YouTube ↗54m

" summary

Describes building a specialized time-series database on top of Apache Kafka to meet high-load requirements for performance, low latency, and storage efficiency. Covers the team’s experience developing and operating this Kafka-based timeseries DB as a bespoke alternative to adding another cache or database.

AI-written summary · about

" chapters (14)

1. 0:00Introduction
2. 1:11Metrics and platform issues
3. 7:09Database log concept
4. 10:22Kafka as reliable log
5. 13:13Kafka data structure
6. 16:27Reading from Kafka
7. 22:44Long-term data storage
8. 25:11Error handling in Kafka
9. 28:01Kafka production setup
10. 30:09Kafka issues and bugs
11. 32:14Scaling Kafka
12. 36:24Kafka disk usage
13. 38:12Conclusion and recommendations
14. 43:16Q&A

" related talks

• 45mApache Kafka as the Single Source of Truth: Usage, Problems, Pain and ConsequencesDm. Emelyanov · HighLoad++ 2024
• 13mKeynote – Jay KrepsJay Kreps · PGConf Silicon Valley 2015
• 38mRe-architecting Apache Kafka for the CloudGwen Shapira · Craft Conference 2021
• 01h 10mApache Kafka: What It Is and How It Will Change Your Application ArchitectureIvan Ponomarev · HolyJS Piter 2020
• 29mApache Kafka and the Next 700 Stream Processing SystemsJay Kreps · Strange Loop 2015
• 41mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia November 2020
• 39mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia Europe 2020
• 01h 00mKafka as a Platform: the Ecosystem from the Ground UpRobin Moffatt · NDC Oslo 2020