Luigi - Big data plumbing library
summary
Luigi is an orchestration library (born at Spotify) used at Crosswise/Oracle Data Cloud to schedule batch pipelines written in Python/Scala/Java across Hadoop, Spark, Pig and EMR; it processes about 1.3 TB/week and manages graphs that grew to roughly 3,000 vertices (visuals now trimmed to ~170). Tasks declare code, parameters and outputs; targets can be files, DB rows, S3 objects or entire EMR/S3 targets. Runtime features include signature-based deduplication to avoid duplicate runs, dynamic graphs, wrapper tasks, resource hints and Jenkins-driven scheduling.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:38About Crosswise (Oracle Data Cloud)
- 1:14What is a luigi?
- 2:27Pipeline - 2014-20
- 3:49Data Flow Framework Requirements
- 5:13Many competitors
- 5:32Key Concepts - Task
- 6:09Key Concepts - Dependencies
- 6:20Key Concepts - Parameters
- 6:59Key Concepts - Output Target
- 8:07Build in features
- 10:16Example - Stock luigi code
- 11:16Example - Crosswise's Luigi code
- 11:52Luigi by Crosswise
- 13:52Visibility
- 14:45Extended Execution
- 15:20Usability
- 16:22Lessons learned
- 18:22Crosswise with and without luigi
related talks
28mDynamic Data Pipelining with Luigi
54mData engineering & DataOps: Ask Me Anything
41mUnifying Diverse Logs in Big Data Systems for Seamless Analysis and Action With OpenSe...
16mReplacing Makefiles with Python and Luigi
47mAll the DataOps, all the paradigms
35mMachine Learning Pipeline using Luigi and Scikit Learn
40mTop 10 data engineering mistakes #bbuzz
01h 06mBest practices when accessing Big Data or any other data!