Automating a Big Data Platform with Jenkins

on YouTube ↗44m

" summary

Sanoma replaces a brittle ETL stack with Jenkins-driven pipelines on Hadoop and Hive, switching to Jenkins in 2012 to schedule and version processing and expose self-service to analysts. The platform runs on ~55 nodes with 650 TB of storage, supports 3,000 jobs a day, ~250 dashboard users and ~40 analysts, and ingests ~25 sources into a few hundred Hive tables. Processing uses Titan for ingestion, Jython-style scripts and MapReduce/Cascading, Yarn, Kafka/Storm for realtime, and ClickView for reporting.

AI-written summary · about

" chapters (8)

1. 0:00Start
2. 23:48Levels of Self-Service
3. 27:01Lease Parameterize Builds
4. 30:33Current State
5. 31:33Use Cases
6. 36:38Main Takeaways
7. 39:20What Are the Dependencies
8. 40:16The Dependency Graph Plugin

" related talks

• 38mOperating Jenkins at ScaleRobert Sandell, Ryan Campbell · Jenkins User Conference US West (JUC San Francisco) 2014
• 38mOperating Jenkins at ScaleRobert Sandell, Ryan Campbell · Jenkins User Conference San Francisco 2014
• 49mData Platform Architecture, Evolution, and PhilosophyJustin Becker, Kurt Brown · @Scale 2014
• 30mA Journey Introducing Jenkins to Enterprise ITDan Cundiff · Jenkins User Conference US West (JUC San Francisco) 2014
• 35mJenkins Scaling and Organization for an Efficient CIAndrea Giardini · DevOps Pro Europe 2019
• 30mA Journey Introducing Jenkins to Enterprise ITDan Cundiff · Jenkins User Conference San Francisco 2014
• 55mContinuous Integration on SteroidsAleksandr Akbashev · HighLoad++ 2016
• 41mBig data as we know it @bol.comDebarshi Basak, Mark Nijenhuis · J-Fall 2015