How Big Data is Powering the Internet of Things (IoT) Revolution

on YouTube ↗43m

" summary

Claims that Big Data pipelines make IoT analytics practical by combining batch and real-time layers; demo builds a Python faker producer, sends simulated sensor events through Amazon Kinesis + S3, processes them with Spark Structured Streaming on Databricks, and visualizes results. Uses 100 simulated warehouse devices, two-hour tumbling windows, finds 28 devices in “red” CO2 range and some countries with max temps above 90°F (Philippines spikes), and surfaces batteries with level 2/1 (three devices at level 2). Streaming runs live, shows input vs processing lag and is used to trigger operational alerts.

AI-written summary · about

" chapters (44)

1. 0:00Intro
2. 0:18Aidas background
3. 1:18Aidas work
4. 2:11Overview
5. 2:43IoT Devices
6. 3:44What is IoT
7. 4:32Benefits of IoT
8. 5:59Challenges of IoT
9. 8:08White Cloud
10. 10:55IoT Helix
11. 12:01Indoor Air Quality Monitoring
12. 12:17Indoor Air Quality Monitoring Example
13. 13:30Analysis
14. 14:33Architecture
15. 15:43Sending Data
16. 17:02ingestion and storage
17. 17:17data generation
18. 17:56data analytics
19. 18:09data bricks
20. 18:34data description
21. 19:18batch processing
22. 20:02big files
23. 20:39read data
24. 21:25sample data
25. 22:11data continuity
26. 23:13data analysis
27. 26:56streaming processing
28. 27:06streaming
29. 27:58streaming vs batch
30. 28:45input vs processing rate
31. 29:51how a stream works
32. 30:25low battery level
33. 30:36even vs received time
34. 31:34maximum day
35. 31:45latest event
36. 32:13reading to memory
37. 32:48creating queries
38. 33:22battery levels
39. 35:41group data example
40. 36:11query
41. 37:33high levels
42. 38:10active queries
43. 38:37historical batch
44. 39:31questions

" related talks

• 31mUnderstanding the Real World: Streaming Analytics for IoTÁlvaro Egea · T3chFest 2016
• 56mTaming IoT Data: Making Sense of Sensors with SQL StreamingHans-Peter Grahsl · Voxxed Days Cluj-Napoca 2019
• 31mFun with SensorsYogi Rampuria · Distributed SQL Summit Asia 2022
• 22mBoosting Traditional Business by IoT and AnalyticsVesa Pirinen · apidays Helsinki 2019
• 24mUnlocking the potential of the Internet of Things (IoT)Cindy Polin · AWS Summit AMER 2020
• 23mKeynote: Intelligent Internet of Things: Start Analyzing Your Global Device Data for Real-TimeAntony Passemard · Embedded Linux Conference + OpenIoT Summit North America 2018
• 36mHarnessing the Full Potential of IoT Data with AWSTom McMeekin · AWS Summit Online 2020
• 20mAutomating Data Science for the IOTRuban Phukan · Data By the Bay 2016