SDKs Are Not Services and What This Means for SREs
summary
Criteo builds an embedded SDK scheduler in Scala to orchestrate hourly Hadoop/MapReduce and Hive jobs across a platform that sees ~1+ billion monthly users and ~0.5 trillion HTTP events/day; the SDK lands in ~200 deployed processes and produces exabytes, but causes high SRE toil (supporting client runtime issues, a six‑month MySQL→MariaDB migration, handoffs, surprising release‑lag bugs). The team pivots to a declarative SQL-as-service DSL that centralizes scheduling, reduces client-side complexity, and avoids embedding stateful infrastructure in libraries.
AI-written summary · about
chapters (18)
- 0:00Start
- 0:43Agenda
- 2:15Deep Thought #1: software engineering is about providing utility
- 5:28Motivation
- 6:28The Early Years
- 7:46The Early Wonder Years
- 12:19New Problems, New System
- 17:01The adoption rate is tepid at best
- 21:22Going further in our analysis
- 29:38A mid-term post mortem
- 31:01This isn't working, part deux
- 32:12The nice properties of SQL and the RDBMS model
- 33:46Scheduling with SQL
- 35:32SOL VS SDK approaches
- 37:45Exporting data with SOL
- 38:47Summary findings
- 39:52Lesson 2
- 40:11Lesson 3
related talks
21mSRE: An incomplete guide to cultural Narnia
27mSublinear Scaling in Practice: The 1k SRE Project
24mSREs are Driving the Evolution of Developer Self-Service
24mHow HashiCorp Implements SRE
29mStory of Seeking Sustainable SRE
32mWhy we skipped SRE and switched to Platform Engineering?
35mSRE -- culture and strategy @ TrustingSocial
14mWeb Apps, through the Lens of SRE (Site Reliability Engineering)