Building a community fountain around your data stream
summary
Uses Apache Kafka for replicated, rewindable delivery, Avro schemas for compact, evolvable messages, and Spark streaming to run batchlike filters from Jupyter. A Dockerized testbed and notebooks are available on GitHub and deploy to mock an astronomical stream; it accepts Avro alerts and stamp images. Target scale is concrete: ZTF ~1,000 alerts per 45s (20–40 GB/night) and LSST ~10,000 per 39s (≈1 TB/night), so throughput and schema evolution are demonstrated.
AI-written summary · about
chapters (8)
- 0:00Streaming data intro
- 3:12Why streaming matters
- 14:31Overcoming streaming fear
- 16:22Inclusive data ecosystems
- 23:25Kafka Avro Spark stack
- 32:17Filtering and brokers
- 35:54Spark streaming analysis
- 38:54Q&A
related talks
50mA Journey in Space with Apache Kafka data streams from NASA
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
29mData Acquisition: Overcoming Challenges
20mBuilding realtime efficient queries in an IOT platform
38mOn Track with Apache Kafka
43mHail Hydrate! From Stream to Lake Using Open Source
36mKafka Streaming, the Pythonic Way
40mStreaming Geospatial Vector Data with Apache Projects