Time Series Job Scheduling at Criteo with Cuttle
summary
Criteo replaced a single Ruby scheduler with Cuttle, a distributed Scala scheduler that runs ~300k jobs/day over ~4PB of logs to produce trillions of records. Jobs operate on hourly partitions with time-series-aware DAGs expressing dependencies via time offsets, support prioritized resource management, backfill/replay with audit trails, and guarantee at-least-once execution (expecting idempotent jobs); the project is open source on GitHub.
AI-written summary · about
chapters (8)
- 0:00Criteo data engineering
- 5:19Time series job scheduling
- 10:37Job dependency challenges
- 15:37Scheduling solutions evolution
- 21:47Centralized scheduling issues
- 27:46Cuttlefish approach
- 29:56Cuttlefish demo
- 44:35Cuttlefish Q&A
related talks
51mStream all the things! - life at the end of the firehose
27mPayment workflow with Ruby and Clickhouse
46mEvents Analysis with Riak, Perl and Erlang at Booking.com
35mProcessing data at scale with Rails
38mRTB and Big Data - where Erlang and Hadoop meet
29mFunctional Programming & NoSQL Design
36mData pipelines engineering made simple with Scala
48m10 Billion a Day,100 Milliseconds Per: Monitoring Real Time Bidding