Building realtime efficient queries in an IOT platform
summary
An IoT pipeline that ingests via WebSocket/CoAP uses Avro/MessagePack, Kafka + Schema Registry and converts rows to Apache Parquet for HDFS, yielding about 30–40% storage savings and batch jobs that run ~3–4x faster. Real‑time processing uses Samza and Spark Streaming (evaluating Flink) with a Samza-based rule engine and Redis for in‑memory datasets. Cassandra holds time series data augmented by SASI secondary index (tested under load but not yet in production) with important caveats about partition keys, sorting and two‑pass reads.
AI-written summary · about
chapters (9)
- 0:00Introduction to IoT platform
- 2:21Backend stack overview
- 3:46Apache Parquet benefits
- 9:23Parquet conversion pipeline
- 11:52Batch vs streaming tradeoffs
- 12:14Cassandra SASI index
- 16:55SASI index use cases
- 17:38Stream processing options
- 19:27Q&A session
related talks
08mReal-Time Data Systems: Speed, Scale, and Resilience
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
56mTaming IoT Data: Making Sense of Sensors with SQL Streaming
31mFun with Sensors
34mBuilding and Scaling Robust Zero-code IoT Streaming Data Pipelines
43mBuilding a community fountain around your data stream
38m7 reasons to use Apache Flink
48mContinuous Optimization for distributed BigData analysis