Tunnel Snakes Rule! Bringing the many worlds of Python together to monitor Melbourne's biggest infra
summary
Claim: a production system ingests 150–200 sensors sending half‑second telemetry and survives a peak of ~1 billion records/month. Built components include device agents (API pollers, FTP server using PyFTPDlib), a Kinesis pipeline + S3 audit AWS Kinesis, stream processing (checkpointed Flink) Apache Flink, time‑series storage InfluxDB, and a Django/Celery/React frontend that uses pandas for live regulatory calculations and PDF reports — deployed and running under real project deadlines.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:15Project Overview
- 1:30Parkville Precinct
- 3:00The problem
- 7:04The plan
- 13:10The build
- 17:38APIs
- 19:37FTP
- 21:34Streaming Analytics
- 24:38Why pandas
related talks
43mTowards a full stack python monitoring+analysis framework
35mPython at Cloudkick
28mDiving into the Deep End: A Python Journey in Water Management
26mFrom Data to Decision: Monitoring and Improving Water Networks using AI/ML
30mPython Platforms for Cyber Security
35mPython at Cloudkick
26mBuilding A Python Service Using Open Source Components That Connect, Transform & Deliver Data In RT
41mThe Snake in the Tar Pit: Complex Systems with Python