A Journey in Space with Apache Kafka data streams from NASA
summary
Reads live NASA GCN Kafka topics with an Apache Kafka client, ingests ~3,000 historical messages, and shows a reproducible GitHub notebook pipeline on the Databricks Intelligence Platform. Builds create streaming table + materialized view SQL pipelines that parse message blobs into columns, compares ingestion approaches (Spark notebooks vs declarative pipelines), cites Photon speedups and P99 100 ms real‑time streaming latency claims, demos plotting (sometimes flaky) and shares code to replicate.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:51NASA gamma-ray burst detection
- 6:09Swift satellite tracking demo
- 11:16Why streaming data matters
- 13:03Kafka for global data distribution
- 16:15Lakehouse architecture intro
- 18:24Cookie sales analysis demo
- 21:59Throughput vs latency
- 24:29Performance benchmarks
- 25:31AI integration in data
- 27:17Connecting to NASA Kafka
- 29:04Reading GCN Kafka messages
- 30:31Streaming pipeline in SQL
- 33:24Governance and lineage
- 35:54AI SQL code generation
- 37:21Genie room setup
- 40:14Gamma-ray burst analysis
- 42:04Space event visualization
- 45:01Conclusion
- 45:43Q&A
related talks
43mBuilding a community fountain around your data stream
29mData Acquisition: Overcoming Challenges
20mKeynote: Apache Kafka and the Rise of the Streaming Platform
36mKafka Streaming, the Pythonic Way
40mStreaming Geospatial Vector Data with Apache Projects
38mOn Track with Apache Kafka
50mBuilding Data Streaming Platform with Apache Kafka
07mTransforming Big Data into Real-Time Product Insights