On the Scalability of the Erlang Virtual Machine and its Term Storage
summary
Benchmarks use the open benchErl suite and real apps to show which VM changes actually move the needle: lock‑free scheduler queues, a new allocator (address-order first carrier best fit), and ETS options (read_concurrency, write_concurrency). Results: Mandelbrot achieves ~25× speedup on 32 cores; many workloads show R15–R17 removing R13’s negative scalability; ETS hash tables stop collapsing while ordered_set still limits scaling and needs redesign. The team runs live bencher demos, ports to Blue Gene, and reports concrete per‑release timings and Numa sensitivity.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:54Overview • Erlang and aspects of its concurrency model
- 4:13Why is Erlang Interesting? • Appropriate for large scale concurrent execution . Concurrency is built-in to the language • not an afterthought or provided via a library . 'Shared nothing by default concurrency model • requires no explicit locking, in principle scales
- 6:37Implementation of Concurrency • Erlang processes are implemented by the VM's runtime system, not by Os threads • Multitasking is preemptive (the VM does its own process switching and scheduling) • Processes use very little memory initially, and switching between processes is very fast • Erlang can handle very large numbers of processes • Some applications use many thousands of them . On multicores, processes can be scheduled in parallel on separate logical cores
- 8:17Starting Processes • The spawn functions create new processes spawn (fun () - ... end) spawn (Module, Function, [Al-An])
- 9:51Erlang's Runtime System (VM) • Handles the basic built-in things • Memory allocation . Garbage collection . Process creation • Message passing • Process scheduling Several possible ways of structuring the RTS . Trade-offs have been studied long ago • unfortunately, on single core architectures
- 11:43Process Local Heap Organization Pros: • Isolation and robustness • Processes can be GC-ed independently • Fast memory deallocation when a process terminates: heaps are used as regions/arenas
- 21:24Lock-free data structures for key components • Optimizations of process tables inside the VM • Lock-free data structures for scheduler queues (starting with R16B) • Optimizations in the area of port locks inside the VM
- 24:37BenchErl: Benchmarking Suite . Started with 14 concurrent programs used at Ericsson for benchmarking the performance of the SMP implementation of Erlang/OTP • Also added some bigger applications
- 42:24Key component of Erlang/OTP Key/value store mechanism Heavily used in applications Supports mnesia Provides shared memory with destructive updates! crucial for parallelization a scalability bottleneck?
- 42:54Erlang Programming in the "real world" dtop
- 46:11AVL Trees Used for ETS tables of type ordered_set Balanced binary search trees
