Visual Flow, open-source ETL tool powered by Apache Spark
summary
Visual Flow is an open‑source, cloud‑native ETL GUI that stitches Apache Spark, Kubernetes and Argo into runnable workflows stored as config maps and workflow templates in Kubernetes and executed as pods. A demo loads two CSVs from S3, unions them and writes to DB2 on Cloud — job deploys in about a minute and succeeds. Claim: common small ETL tasks go from 15 minutes in Visual Flow vs ~2 hours with manual Scala/Spark tooling; supports S3, IBM Cloud Object Storage, Elasticsearch, DB2, multi‑tenant namespaces, role-based access, and pipeline scheduling/alerts.
AI-written summary · about
chapters (18)
- 0:00Introduction to Visual Flow
- 1:12What is Visual Flow
- 2:17Why Visual Flow was created
- 3:42Visual Flow vs manual ETL
- 4:47How Visual Flow works
- 5:30Supported data sources
- 5:51User roles in Visual Flow
- 6:34Project structure and parameters
- 7:17Creating pipelines in IDE
- 8:21Visual Flow development roadmap
- 10:11Summary and benefits
- 11:17Live demo introduction
- 11:59Demo setup and job creation
- 13:23Navigating Visual Flow UI
- 14:25Configuring job stages
- 17:54Saving and running the job
- 18:36Job execution and results
- 20:09Q&A and contact information
related talks
30mCloudflow: Spark, Flink, and Akka Working together on Kubernetes
17mDatabricks LakeFlow: A Unified, Intelligent Solution for Data Engineering
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
27mVisualizing Cloud Systems
22mDo Flink on Web with FLOW
01h 28mA Guide to the Modern Data Stack
39mComplex Data Transformations Made Easy
38mUnleashing the serverless data pipelines