Deploying machine learning pipelines to production
summary
Cloud pipeline design uses Apache Avro for compact, schema-evolving packets (including binary image cutouts) and Apache Kafka with mirror-maker for streaming and hub replication. Development runs in Docker containers and Jupyter notebooks so analysts can iterate locally; production archives Avro files (~75 GB/night) and the live system handles about 1.2 million alerts per night. End-to-end pipeline reaches consumers in ~20 minutes (packaging/transfer ~4 seconds) and allows stream replay for debugging and reprocessing.
AI-written summary · about
chapters (9)
- 0:00Start
- 5:03Data Scientist wish list
- 12:03Zwicky Transient Facility
- 12:54Large Synoptic Survey Telescope
- 17:53Traditional Format
- 20:19Data formatting: Apache Avro
- 24:53Traditional Method
- 25:57Data access: Apache Kafka
- 33:30Putting it all together: ZTF Alert Distribution System (ZADS)
related talks
20mShip your Machine Learning Application #bbuzz
35mHow to deploy machine learning models to production (frequently and safely)
49mOrchestrating data and ML workflows with Apache Airflow
57mDevOps for Machine Learning
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
17mR & Python playing nice, in Production
29mAI-Powered Anomaly Detection in Cloud Data Pipelines
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow