How Fast Can You Parse 1 Billion Rows in Java? – Insane Speed Test
summary
Challenge turns a 4 minute 50 second baseline into about 1.5 seconds to parse 1 billion weather rows using Java, demonstrating concrete techniques: memory-mapped files, Graal native image and Epsilon GC, direct access via sun.misc.Unsafe, SWAR/“SIMD-as-register” bit tricks and branchless parsing, a custom forward‑probing hash table, multi‑slice work stealing and triple scanners per core, plus perf-driven tuning. Resulting tool parses the 16 GB file with ~100 billion instructions (≈100 instructions per record), the micro‑optimizations and kernel unmap workaround all demonstrably improving throughput.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:00The challenge
- 4:07Watch, learn, adopt, experiment
- 5:35Which JVM?
- 7:03Temperature as integer
- 8:01Memory mapped files
- 9:35Getting unsafe
- 11:04SWAR
- 14:08Stringless
- 14:55Branchless programming
- 16:52Parse the temperature
- 24:22Exploit the data
- 25:22Mechanical sympathy
- 26:32Keeping track
- 34:59Remember this?
- 36:40Results
- 37:05Summary
- 38:39Outro
related talks
42mJava, How Fast Can You Parse 1 Billion Rows of Weather Data?
02h 57mPushing Java to the Limits: Processing a Billion Rows in under 2 Seconds
47mConquering a Billion Rows with Modern Java Concurrency
47mThe First 80% of Reading One Billion Rows Fast Enough
49m1BRC – Nerd Sniping the Java Community
46mLessons from 1BRC: Tricks and Approaches to Performance Optimization
51mSecrets of top performance in Java revealed
47mUltralow Latency with Java 11 and Terabytes of Data