What is multimodal RAG, and can we build a village with it?
summary
Multimodal RAG pipeline that injects text images and audio into prompts to power living NPCs in a demo orc village called Little Minos Morgle; backend uses VV8 vector DB, Spring + Langchain4j (Kotlin), Unity frontend, local models (Olama Olama 3.1 8B, LAVA, CLIP/CLAP) and a small Python service for audio. NPCs decide actions every ~30s, creative then structured JSON prompt fixes hallucinations (had to add “strictly”), model swapping stalls on an 8GB GPU, overall demo works with latency and formatting caveats.
AI-written summary · about
chapters (9)
- 0:00Introduction and origin
- 2:30Understanding RAG
- 3:43Defining multimodal RAG
- 7:21AI models and architecture
- 10:01Building the orc village
- 13:36NPC logic and movement
- 20:04Text-based interaction
- 24:31Image-based RAG integration
- 32:39Audio RAG and conclusion
related talks
43mWhat is multimodal RAG, and can we build a village with it?
41mWhat is multimodal RAG, and can we build a village with it?
50mWhat is multimodal RAG, and can we build a village with it?
31mBuild an Agentic RAG system using Langchain, Ollama and Milvus
31mOrchestrating Multimodal RAG With Agentic Workflows
42mTo RAG or not to RAG
48mCrafting your own RAG system: Leveraging 30+ LLMs for enhanced performance
41mFrom RAGs to riches: Build an AI document interrogation app in 30 mins