Kronk: Hardware Accelerated Local Inference
summary
Kronk is an SDK that runs Llama CPP-based GGUF models locally on Metal/CUDA/Vulkan so Go programs do inference without Python, delivering an SDK plus a test model server (KMS). It uses EMA via PureGo and FFI to load libraries dynamically, ships examples (tic‑tac‑toe and an agent that became Player2), and demos work live: a 0.5 GB model runs CPU, small agents fit under 8 GB, and large models cost ~10 GB per execution slot for 128k context. Clone and run make example question to reproduce the demo.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 6:29Why Kron replaces servers
- 19:43Live demo
- 21:50Processing model requests
- 37:09Tokenization and GPU setup
- 46:01Token generation explained
- 55:30Caching benefits
- 1:00:04Closing
related talks
32mCyfra: Scala on GPU
29mThe Go-to Language for AI: Opportunities and Challenges
37mA Funny Thing Happened On The Way To Reimplementing AlphaGo
25mLeveraging Wasm for Portable AI Inference Across GPUs, CPUs, OS & Cloud-Nativ...
21mXorbits Inference: Model Serving Made Easy
36mAccelerating High-Performance Machine Learning at Scale i...
19mThe Small Model Infrastructure Nobody Built (So We Did)
21mOptimizing Model Serving on Kubernetes With Model Streaming