Friendship of Data with Cloud: Close friends or Best friends?
summary
Shows an end-to-end AWS pipeline: EC2 + Kafka + Spark pipeline with Kinesis and Lambda for streaming, Aurora/MongoDB storage and Plotly/Dash visualization; demonstrates real-time Twitter ingestion and claims lower cost and easier scaling versus on‑prem (cites Hotstar scale needs around 13 million viewers as motivation). Uses NiFi/Kafka for ingestion, Spark ML for processing, and argues security remains a 50/50 concern. Concludes data and cloud are close friends today and could become best friends if trust and services improve.
AI-written summary · about
chapters (9)
- 0:00Data and cloud friendship
- 1:16Agenda and cloud basics
- 3:26On-prem systems issues
- 6:16Shift to cloud technologies
- 7:19Cloud tools for data problems
- 9:57End-to-end ML with cloud
- 11:20Real-time site monitoring
- 14:18Close vs best friends in data-cloud
- 16:26Conclusion and future
related talks
35mStreaming and Real-time Analytics
35mStreaming and Real-Time Analytics
41mOn Data: Sky above, sand below, peace within
29mSimplifying data flows
29mGrowing insights through hybrid data architectures
47mBuilding Modern Data Streaming Apps with Open Source
53mDefine data streaming architectures & solutions to fit your needs
42mTransform your data approach: Develop a modern data strategy