Background Jobs with Elixir: Oban in the Real World
summary
Company runs Oban on four Fly nodes against a managed Postgres and faces ten million jobs a day workloads; queues are paused to stop churn, then auto_vacuum settings are tuned and an immediate reindex concurrently (≈20 minutes) clears heavy index bloat caused by MVCC churn from multiple updates per job. Patterns include many granular queues, self‑enqueuing workers and uniqueness constraints, and the hard lesson is monitoring PG stats and vacuum health, not just job telemetry.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:52Motivation for Oban at Fairing
- 5:00Monday production incident
- 10:47Postgres MVCC and vacuuming
- 13:55Root cause: bloated jobs table
- 15:23Solving incident and preventing recurrence
- 17:30Tuning auto vacuum and pruner
- 19:36Separating connections and queue granularity
- 22:29Self-enqueuing workers and cron alternatives
- 26:01Batching jobs and Oban Pro
- 27:47Observability gap in monitoring
- 30:37Four dimensions of Oban monitoring
- 34:30Key takeaways
related talks
26mScaling Oban Applications
25mMigrating from C# to Elixir
47mGood behaviour: Cultivating Healthy Elixir Teams & Codebases
37mUsing Elixir for scheduling cloudnative DAG workflows
24mGoing CPU Bound
45mElixir: une potion pour mixer complexité et concurrence en temps réel
40mDistributing Work With Queues and GCP PubSub
42mThe Soul of Erlang and Elixir