How We Built Tools That Scale to Millions of Lines of Code
summary
Twitter faces tens of millions of Scala lines organized into tens of thousands of targets and builds a lightweight semantic pipeline instead of reusing the full compiler: a SemanticDB protobuf schema plus a scalac plugin emits per-file semantic payloads, those payloads feed a queryable SQLite index (≈500 MB per million lines) with single- or double-digit ms lookups, exposed via a company-wide language server and used by Metals/Scalafix/meta-doc and distributed rewrite jobs — experimental results show it scales and works.
AI-written summary · about
chapters (26)
- 0:00Introduction
- 2:26The Problem
- 7:10The Solution
- 11:52Overview
- 15:22Taystee
- 16:20Scala Meta
- 17:18Semantic DB
- 18:12Text Document
- 19:02Semantic DB Example
- 20:19Metabeam
- 20:48Symbols
- 21:18Currencies
- 22:05Scala Sphere in Crime
- 22:55Examples
- 23:17MetaDoc
- 23:41MethodDoc
- 25:44Metals
- 26:37Metal
- 27:33Code Actions
- 28:40ScalaFix
- 30:25Companywide Semantic Index
- 33:02Experimental IntelliJ Plugin
- 34:07Fabricator
- 34:57Evolution
- 36:17Summary
- 37:13Questions
related talks
57mBuilding code analysis tools at Twitter
19mExploring Scala Tooling ecosystem
50mMove fast and fix things
29mExploring Scala Tooling ecosystem
32mBetter Scala builds with the Mill build tool
49mBuilding Safe, More Testable Scala Production Systems
01h 29mScalafix Workshop
39mUnleashing Scalafix potential with custom rules