How AI Ate My Website
summary
Personal website becomes a searchable, multimodal knowledge base: 2,000 articles, 100+ videos, ~500,000 words and 6,000 minutes of video plus ~400 PDFs get converted via speech-to-text, diarization, FFmpeg thumbnails, vision models and embeddings into chunks, indexed with a custom retriever and ranked by a cross-encoder. The live service Ask@lukew.com surfacing inline citations, video timestamps, suggested questions and images runs for ~1.5 years, improves with cleaning and fine-tuning, and yields answers that mirror the author’s voice rather than generic chat output.
AI-written summary · about
chapters (13)
- 0:00Introduction to AI
- 1:38How AI ate the website
- 3:08AI models on the web
- 6:20Capabilities and ingredients
- 9:20New interaction models
- 12:08Chatbots vs conversational UI
- 15:00Designing for AI
- 20:17Processing content for AI
- 26:36The virtuous loop
- 32:04Content strategies and cleaning
- 34:56Multimodal AI in action
- 40:26Audience Q&A
- 47:42Future of AI and conclusion
related talks
31mBring AI based Search to Your Website
43mHarnessing AI to strengthen trustworthy information
37mBest AI Tools for Web Developers / How to Use AI in Web Development
19mHow to get clean web data for chatbots and LLMs
34mJavaScript-Powered AI: Crafting Intelligent Systems with LLMs and Vector Databases
30mBeyond Keywords: Building an AI-Powered Search That Reads Your Mind
20mAnswers. Agents. AGI.
28mAI’s Next Great Challenge: How to create AI that truly understands us?