Prefect Workflows for Scaling Acoustic Fisheries Survey Pipelines

on YouTube ↗29m

" summary

Claims a working open-source ecoflow package that uses Prefect 2.0 for orchestration, YAML-configured pipelines, and native dask cluster parallelism. Shows conversion of raw sonar to cloud-optimized zarr, runs 10 files (≈300 MB) in ~25s on five workers, and targets 250 TB legacy datasets. Demo covers S3 input, EC2/ECS/systemd deployment, blocks for credentials, and a basic rule engine — concrete code, configs, and a local Prefect demo that completes successfully.

AI-written summary · about

" chapters (9)

1. 0:00EchoFlow introduction
2. 3:17Why EchoFlow needed
3. 7:54Sonar data workflow
4. 9:20Prefect over Airflow
5. 10:44Prefect architecture
6. 13:35EchoFlow components
7. 17:34EchoFlow deployment
8. 20:23Live demo setup
9. 27:55Q&A

" related talks

• 17mGoing with the flow? Apache Airflow for operational-quality scientific workflowsPatrick Sunter, Michael Pegios, Daehyok Shin · PyCon AU 2025
• 39mSell cron, buy Airflow: Modern data pipelines in financeJames Meickle · PyData DC 2018
• 26mGo with the Flow: Automating Your Workflows with AirflowLeo Guinan · PyOhio 2018
• 01h 59mIntroduction to orchestration ETL processes with Apache AirflowMiroslav Tomic · DSC Europe 23
• 21mScaling and automating R workflows with Kubernetes and AirflowIsaac Florence · rstudio::conf(2022)
• 37mBatch Processing Tekton Pipelines in OpenShiftMartin Bacovsky · DevConf.CZ 2023
• 42mHow I learned to time travel, or, data pipelining and scheduling with AirflowLaura Lorenz · PyData DC 2016
• 21mScaling and automating R workflows with Kubernetes and AirflowIsaac Florence · rstudio::conf 2019