Analysing Public Kafka Data from NASA Satellites
summary
Shows how NASA’s GCN public Kafka streams feed a Databricks Lakehouse using Apache Kafka, Apache Spark (Spark 4 declarative pipelines), Unity Catalog and MLflow; implements a SQL-only pipeline with streaming tables and materialized views despite OIDC/Kafka client friction. The demo ingests ~3,333 recent events, parses message blobs into columns (including RA/DEC coordinates and merit score), produces an interactive sky plot and exposes natural-language queries via Genie. Code and blog are published and the pipeline runs end-to-end.
AI-written summary · about
chapters (15)
- 0:00Introduction and speaker background
- 3:15Photography and cosmic events
- 7:53NASA Swift satellite and gamma ray bursts
- 10:19Neutrino data and multimodal research
- 11:46Public Kafka data from NASA
- 15:15Lakehouse platform overview
- 16:59Demo setup and data pipeline design
- 19:09Streaming tables and materialized views
- 22:00Kafka connection and SQL setup
- 23:26Ingesting and processing NASA data
- 26:16Data transformation and pivoting
- 28:44SQL analytics on cosmic events
- 31:15Natural language queries with Genie
- 34:07Visualizing cosmic event coordinates
- 37:02Conclusion and resources
related talks
43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
47mBuilding Modern Data Streaming Apps with Open Source
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
31mOne Data Pipeline to Rule Them All
31mDIY: Data streaming platform with Kafka and Java
49mEnriching Kafka Applications With Real-time Contextual Data
46mKafka Streams at the Big Data Crossroads at Lightning Speed