Pyrobuf and the Tyranny of Speed
summary
Pyrobuf is a Cython implementation of Protobuf that generates a custom C-extension per .proto file and claims 2–4× vs C++ backend and 20–40× vs pure Python serialization speed, requiring a C compiler and supporting Python 2.7 and 3.5. Benchmarks use pytest-benchmark. Practical optimizations shown include bypassing __init__ for fast instantiation, avoiding string slicing by passing offsets, and skipping type-guard checks to shave single-digit percentages. CaptainProto serializes ~6× faster and deserializes ~3× faster but is ~4× slower end-to-end. Tradeoffs include bugs, security, and compatibility costs.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:53About Me
- 1:22Outline
- 1:48What is the Problem?
- 3:42XML and JSON
- 4:13Pickle
- 6:49Backwards and Forwards Compatibility
- 11:17General Advice on Optimising Code
- 12:29Static Type Declarations
- 14:03First You Need to Measure It
- 16:31Cython Fast Instantiation
- 18:02Avoid unnecessary Slicing
- 19:56Sacrifice Correctness!
- 22:09How Does Cap'n Proto Achieve this Miracle?
- 23:30Performance is a Race!
related talks
28mEffective Protobuf: Everything You Wanted To Know, But Never Dared To Ask
22mPyPy in production
22mPyPy and the future of the Python ecosystem
57mPython and PyPy performance (not) for dummies
17mHigh Performance Python Microservice Communication
31mPumping up Python modules using Rust
33mA low latency deepdive of Python with Cython
01h 21mAccelerating Python With C++