Using Native CPU Vectorized Instructions for 60% Speedup on Lucene's KNN Vector Search
summary
Lucene’s dot‑product hotspot gets rewritten in native SIMD C and delivers a production 60% latency reduction on ARM, with microbenchmarks showing up to 10–13x (Graviton2) and 4–6x (Graviton3) throughput for large embeddings. Implementation ships as a single binary containing SVE and NEON intrinsics plus a scalar fallback, with a runtime resolver that picks the best function pointer. Major pitfall is copy overhead and GC interaction unless using linker.option.critical=true on JDK22, which removes copies but briefly pauses GC.
AI-written summary · about
chapters (17)
- 0:00SIMD and vector search intro
- 1:27Why SIMD matters for search
- 4:15Java SIMD on ARM issues
- 7:00Neon and SVE extensions
- 8:03Native vectorization solution
- 10:08Compile-time details
- 11:13Runtime hardware detection
- 14:26Benefits of native vectorization
- 16:10Benchmark performance impact
- 17:12Indexing slowdown cause
- 19:40Java 22 memory copy fix
- 22:28GC pause considerations
- 23:32Production 60% latency gain
- 24:59Lucene SIMD approaches
- 27:05Enabling native SIMD in Lucene
- 29:13OpenSearch and community benefits
- 34:06Q&A
related talks
46mAccelerating OpenSearch K-NN With FAISS, Leveraging AVX-512 for a...
22mComparing vector implementations in generic databases
43mOpenSearch and the Path to a Trillion-scale Vector Database
21mPerformance Tuning Apache Solr for Dense Vectors
28mThe Memory Efficient Indexing for Vector Databases
32mUnder the Hood: Key Algorithms Powering Vector Search
28mA Deep Dive into Python-Powered Precision and Scalability
26mOpenSearch and the Path to a Trillion-s...