Streaming Visualisation
summary
Three blueprints show concrete ways to visualize fast data: (1) persist to a fast NoSQL store (Elasticsearch, Cassandra or in-memory Redis) but watch polling limits (UIs refresh ~10–15s); Redis wins for many-read, ephemeral maps like live flight positions. (2) stream directly to consumers via Kafka Connect to Slack/REST dashboards or WebSockets for near-immediate updates. (3) run stream analytics with KSQL/Kafka Streams and push results into Arcadia Data for live dashboards — demo with Twitter shows ~2s end-to-end latency and working code on GitHub.
AI-written summary · about
chapters (16)
- 0:00Intro
- 4:20Timely decisions require new data in minutes
- 5:59Keep the data in motion...
- 9:03Reference Architecture Modem Data Analytics Solutions
- 14:24Date-in-Motion Ecosystem
- 16:02Apache Kafka - A Streaming Platform
- 24:34Visualization: many many options! But do they support Streaming Data?
- 26:54Fast datastore with regular poling from consumer
- 28:371: Elasticsearch/Kibana
- 30:182: Redis NoSQL & Custom Web
- 31:122. Demo Redis NoSQL & Custom Web
- 34:37Direct Streaming to the Consumer
- 40:57Streaming SQL Result to Consumer
- 41:281: KSQL and Arcadia Data
- 42:181: Demo KSQL and Arcadia Data
- 48:193. Spark Streaming & Oracle Stream Analytics
related talks
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
41mFast Data - an Overview of the Concepts, Architecture and Technology of Streaming Analytics
38mLive-Coding a Dashboard with KSQL, Python & JavaScript
56mReal Time Data Pipelines
03mVisualizing data with JavaScript at Twitter
27mScalable Data Visualization
41mVisualizing big data
07mTransforming Big Data into Real-Time Product Insights