~/talk

Orchestrating Data Pipelines at Scale with Temporal

on YouTube 41m
Talkat Replay 2026 · 2026-05-05..05-07· [data-engineering, distributed-systems]

speakers

name:Zainab Merchant
links:none on file
name:Madhav Sodhani
links:none on file

resources

video:https://youtu.be/oh7NiAS_qAA
slides:none on file

summary

Fivetran runs over 5 million daily syncs on Temporal workflows, migrates from regionally fragmented schedulers to a worker-pool model, and cuts job startup latency from 25s to 5s. The migration uses Temporal Cloud, a custom operator for long‑running jobs and bin‑packing, and a custom gRPC gateway for hybrid deployments that authenticates agents and routes requests by namespace. Metrics tracked include schedule‑start latency, task depth, worker utilization and cost dashboards; gradual rollout and namespace splitting resolve non‑determinism and scaling issues.

AI-written summary · about

chapters (12)

  1. 0:00Introduction to talk
  2. 1:06Fiverr’s data platform overview
  3. 7:15Why Fiverr adopted Temporal
  4. 9:48Moving from legacy schedulers
  5. 17:12Implementation phases of Temporal
  6. 21:01Pipeline workflow design
  7. 27:21Handling complex pipelines
  8. 30:53Metrics and monitoring during migration
  9. 32:19Challenges and lessons learned
  10. 35:10Hybrid deployment with Temporal
  11. 38:40Future plans and next steps
  12. 40:08Q&A session

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.