Search Engineering Reimagined: Leveraging LLM in ML Workflows
summary
A web3 search engine indexes hundreds of millions of Discord Telegram and Twitter posts to generate trading and marketing signals using embeddings as latent-vector search and clustering, plus a narrative-tracker that auto-generates story clusters and a classifier whose outputs correlate with trade signals. The workflow synthesizes domain data with LLMs, runs SFT and parameter-efficient tuning, then distills to smaller fast models — a pattern called generative ML engineering that yields state-of-the-art embeddings without extra user data. A decentralized experiment called open kle on benser network optimizes DCG via community mining and automated LLM verification rather than central ML training.
AI-written summary · about
chapters (10)
- 0:00Introduction and background
- 1:15Myths about big tech and LLMs
- 3:03Embedding models and common sense
- 5:08Generative machine learning engineering
- 6:32Narrative tracking with LLMs
- 7:37Sentiment analysis for trading
- 8:43Data acquisition challenges
- 10:30Decentralized data indexing
- 13:42Community-driven search economy
- 14:24Q&A and discussion
related talks
49mLLM—Search Engine Killer?
46mLLM—Search Engine Killer?
24mConversational Search with LLMs
51mQuery Understanding With Large Language Models: Techniques and...
26mRevolutionizing Search: Unveiling Conversational Search
01h 30mUsing LLMs to improve your Search Engine
20mTransforming the Search Industry with LLM Using Source Frontier
30mBeyond Keywords: Building an AI-Powered Search That Reads Your Mind