Designing spaCy: Industrial-strength NLP
summary
spaCy is built as a C extension with a Python API and implemented in Cython from the start, enabling GIL-free multithreading and a compact memory‑managed core rather than a separate C++ codebase. The transition parser uses an averaged perceptron with about 9 million sparse features, token objects keep offsets plus a space flag to avoid copying, and word vectors and integer IDs are exposed for fast downstream work; measured throughput claims the fastest words‑per‑second performance and it runs in production (e.g., Reddit demo on ~90 cores).
AI-written summary · about
chapters (9)
- 0:00Introduction to spaCy
- 2:48Academia to industrial NLP
- 3:51Design philosophy
- 7:51Text pipeline & tokenization
- 12:10Dependency parsing
- 16:08C-level data structures
- 20:11Python API design
- 25:53Word vectors & future
- 36:53Q&A on language support
related talks
01h 27mTips, Tricks and Topics in Text Analysis
25mInventing Curriculum using Python and spaCy
46mSPy (Static Python) lang: fast as C, Pythonic as Python
39mPushing Cython to its Limits in Scikit-learn
01h 29mTricks, tips and topics in Text Analysis
01h 14mIntroduction to NLP
21m🤖 Transformer-based NLP Pipelines with SpaCy v3
33mSPy (Static Python) lang: fast as C, Pythonic as Python