What is multimodal RAG, and can we build a village with it?
summary
A working demo builds a living NPC village called Little Minos Morgle using multimodal RAG that injects text, image and audio embeddings at runtime; it uses V8 vector DB, a Spring Boot backend with Langchain4j, Unity front end, and local models like Llama 3.1 via Olama, Lava for vision, CLIP/CLAP embeddings and a bit of Python. NPCs move, converse and consult retrieved docs; a two‑pass JSON validator enforces structured actions. Image and audio search are functional but noisy, and model upgrades easily break the pipeline.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 3:19What is RAG
- 8:15AI and LLM industry
- 8:58Multimodal RAG systems
- 11:07Tolkien village demo
- 14:18Technical architecture
- 15:42NPCs with LLMs
- 19:23Validation agents
- 20:46Text-based RAG with orcs
- 24:54Image-based RAG workflows
- 28:55Orc painter interaction
- 34:41Searching paintings
- 35:02Audio-based RAG
- 38:17Metal music with LLMs
- 41:11Conclusion & Q&A
related talks
41mWhat is multimodal RAG, and can we build a village with it?
40mWhat is multimodal RAG, and can we build a village with it?
50mWhat is multimodal RAG, and can we build a village with it?
31mBuild an Agentic RAG system using Langchain, Ollama and Milvus
42mTo RAG or not to RAG
31mOrchestrating Multimodal RAG With Agentic Workflows
48mCrafting your own RAG system: Leveraging 30+ LLMs for enhanced performance
48mCrafting your own RAG system: Leveraging 30+ LLMs for enhanced performance