Piping Rain in Python - Adventures in a minute latency Radar Pipeline
summary
A passive radar pipeline is built from an SDR and Intel NUC feeding images into S3 and DynamoDB, processed by small Python services (Flask, OpenCV, ImageMagick) with cron-driven jobs and Twitter/WhatsApp bots. It nowcasts an hour ahead in about four seconds, runs on EC2 VMs with packer/ansible deployment, and achieves roughly 70% uptime while serving about 5000 regular users on a few hundred dollars a month. Key failures include S3 event inconsistencies with Zappa and DynamoDB throughput surprises.
AI-written summary · about
chapters (14)
- 0:00Piping Rain in Python - Adventures in a minute latency Radar Pipeline
- 1:36The problem
- 6:03Some semblance of Hardware
- 7:55The hardware on site
- 9:251st attempts with Serverless
- 10:32Running on some EC2 Boxes
- 12:36Cron jobs as data pipelines
- 14:54Multi-S3 bucket raw/processed data storage
- 16:05DynamoDB Sparse Array storage
- 19:02Issues / DevOops - Sensor Site Gotchas
- 20:36Issues / DevOops - Serverless Gotchas
- 24:10Issues / DevOops - Cronjobs and Bot Fights
- 25:22Make it work, then make it better
- 26:08Ask questions in the chat?
related talks
41mReal Time Architectures Real Fast
43mTowards a full stack python monitoring+analysis framework
53mPractical Pipelines A Houseplant Soil Alerting System with ksqlDB
45mResilient Predictive Data Pipelines
41mDeploying machine learning pipelines to production
32m7 steps to build-your-own data pipeline - for day 1 of your startup
17mR & Python playing nice, in Production
44mCreating Python Data Pipelines in the Cloud