Speedy Scala Builds at Databricks

on YouTube ↗32m

" summary

Databricks migrates large Scala mono-repos onto Bazel-style builds, yielding concrete wins: test runs drop from ~150 minutes to ~25 minutes via sandboxed parallelism and up to 96x parallelism; CI compile time cuts 7x with a distributed build cache; cross-building generates per-version modules so upgrades (Scala 2.10→2.12) take about a quarter person-year and compiler speed doubles; remote caching and cloud dev boxes make clean builds effectively instant and give 5–6x faster tests and 1–2 second cold-start developer cycles.

AI-written summary · about

" chapters (8)

1. 0:00Introduction and team background
2. 1:49Why speed up Scala builds
3. 3:14Migration from SBT to Bazeel
4. 8:20Bazeel adoption and test time improvements
5. 10:49Dependency management in Bazeel
6. 16:14Cross-building Scala versions
7. 22:13Compiler performance optimizations
8. 30:40Summary and conclusion

" related talks

• 46mBuilding Scala with BazelNatan Silnitsky · Scala Days Lausanne 2019
• 37mScala with BazelIttai Zeidman, Ahir Reddy, Oscar Boykin · BazelCon 2017
• 08mLightning Talk – Speedy Scala Builds with Bazel at DatabricksAhir Reddy · BazelCon 2019
• 48mMassively Parallel Distributed Scala Compilation And YouStu Hood · Scala Days Lausanne 2019
• 08mScala, Bazel and VirtusLabKrzysztof Romanowski · BazelCon 2024
• 21mReasonable Scala CompilerEugene Burmako · Scale By the Bay 2017
• 43msbt 2 0： go bigEugene Yokota · Scala Days 2025 Lausanne
• 38mScaling ScalaJulien Tournay, Filipe Regadas · Scala Love in the City 2021