Rethinking How We're Linking: What to do when speeding...
summary
Claim: statically linking native extensions into the Python runtime cuts import overhead at very large scale. Implementation uses linkers/loaders, NM and llvm‑objcopy symbol renaming to avoid collisions, LLDB and call-profile tools to find DL_open hotspots, and PyImport_AppendInittab to embed modules. At Meta this lands ~2 megaw savings for one app, ~990,000 rollouts, and saved ~5,000 hours/week of build time; importing thousands (≈8,000) tiny C extensions becomes far faster, though small CLIs can see larger single-binary startup and relocation tradeoffs.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:25Static linking C extensions
- 2:07Python C extension example
- 9:55Python vs C speed
- 19:17Slow imports in C extensions
- 22:08Static vs dynamic linking
- 29:15Embedding C extensions
- 36:22Performance impact of embedding
- 42:55Symbol collisions in linking
- 50:51Q&A
related talks
10mArchitecting Native Python Extension Modules
31mPumping up Python modules using Rust
30mLazy imports and the art of interpreter procrastination
01h 03mA tour of CPython's runtime
59mMyths and fairy tales around Python performance
27mNo time to idle about: Profiling import time in Python
25mHow to bring your C++ project to Python land
06mLessons learned from using the C Foreign Function Interface