Real Time Architectures Real Fast
summary
Claims a workable sub-1s end-to-end latency for live ADSB aircraft tracking using entirely OSS; pipeline is Raspberry Pi + SDR decoder feeding CFKA, Click House with materialized views and Jupyter for queries; aircraft send up to 50 messages/sec, and measured latencies run ~300–500 ms under load though a Cloudflare outage causes weeks‑old gaps in the demo; explores tiered storage and discless CFKA (adds ~200–300 ms) and shows the pipeline mostly works.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:52Live aircraft tracking demo
- 4:00Real-time latency budget
- 8:10Data pipeline overview
- 11:04ClickHouse vs PostgreSQL
- 14:58Kafka in streaming
- 16:43Kafka and ClickHouse demo
- 19:58Optimizing with materialized views
- 23:54Scaling for production
- 26:20Kafka cost strategies
- 28:46Tiered storage in Kafka
- 33:20Key takeaways
- 34:23Resources and GitHub
- 34:45Q&A
related talks
28mPiping Rain in Python - Adventures in a minute latency Radar Pipeline
23m3 Second End to End Latency At Scale
47mA Frontend Server, Front to Back
21mThe challenges of deploying Apple's Low Latency HLS In Real Life
28mRock around the clock (synchronization
18mVideo @Scale 2022 LIVE Q&A #1
01h 45mRun Frontier AI at Home
16mCDN Challenges of HTTP-based Low Latency Live Streaming Delivery