JVM in the Age of AI: Babylon, Valhalla, TornadoVM and friends
summary
JVM inference landscape is practical and fragmentary: CPU-only implementations like llama.cpp = ~700 lines and its Java reimplementation llama.java ≈ 2,000 lines show models running fast on CPU, while GraalVM runs Truffle Python but native C extensions need per-package recompilation so compatibility is hit-or-miss. Valhalla brings 16-bit floats to cut memory, Project Panama and Vector API speed foreign memory and SIMD, and TornadoVM compiles task graphs to CPU/GPU backends — recent Llama forks run on these stacks, though tooling and device transfer remain the main bottlenecks.
AI-written summary · about
chapters (14)
- 0:00Introduction to talk
- 2:58What is model inference
- 6:11GraalVM and Truffle
- 11:55Python in Java with GraalVM
- 15:49Project Valhalla and half floats
- 18:39JNI and Project Panama
- 21:48Running LLMs on CPU
- 27:10Vector API and SIMD
- 30:57GPU programming basics
- 33:46JCUDA and OpenCL limitations
- 37:39TornadoVM introduction
- 41:14Project Babylon and HAT
- 45:08Future of JVM and AI
- 46:53Q&A and conclusion
related talks
49mJVM in the Age of AI: Babylon, Valhalla, TornadoVM and friends
47mPractical LLM Inference in Modern Java
44mJava for AI
51mPractical LLM Inference in Modern Java
50mOne VM to Rule Them All
51mYour next JVM: Panama, Valhalla, Metropolis
38mThe Era of AAP: Ai Augmented Programming using only Java
59mGraalVM: Run Programs Faster Anywhere