What is multimodal RAG, and can we build a village with it?
summary
Multimodal RAG combines text, images and audio embeddings to enrich prompts so NPCs query a local knowledge base and produce grounded responses; implemented with Langchain4j, a VV8 vector store, Unity front end, Spring/Kotlin backend, Llama 3 8B for generation and LAVA/vision + CLAP audio models (Python) for multimodal transcription. A small Mordor village of moving, goal-driven orcs proves the pipeline: image→transcription→embed, audio→embed→retrieve, validator prompts force strict JSON and keep NPC actions coherent while everything runs locally.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:22What is RAG
- 7:39Fine-tuning vs runtime RAG
- 8:00Multimodal RAG components
- 10:07Building village in Mordor
- 13:59Live demo village
- 15:22NPCs decision making
- 21:00Text RAG villagers
- 24:34Image RAG painter
- 29:42Shopkeeper UI flows
- 33:12Audio RAG metalhead
- 39:20Conclusion
related talks
43mWhat is multimodal RAG, and can we build a village with it?
40mWhat is multimodal RAG, and can we build a village with it?
50mWhat is multimodal RAG, and can we build a village with it?
31mOrchestrating Multimodal RAG With Agentic Workflows
42mTo RAG or not to RAG
31mBuild an Agentic RAG system using Langchain, Ollama and Milvus
48mCrafting your own RAG system: Leveraging 30+ LLMs for enhanced performance
22mOrchestrating Multimodal RAG With Agentic Workflows With OpenSearch