Thomas Hazel, ChaosSearch | Supercloud 4
summary
ChaosSearch describes a concrete stack: streaming logs into object stores (S3/GCS) into Chaos LakeDB that auto-discovers and indexes schemas, exposes Elastic APIs and SQL via Trino, and scales customers from terabytes to 300 terabytes per day. The platform integrates public LLMs (OpenAI/ChatGPT, Vertex AI, Amazon) as a reasoning layer and asks the LLM to craft queries rather than returning LLM answers, which lowers cost and reduces hallucination while delivering a conversational AI assistant that runs real queries on the indexed data.
AI-written summary · about
chapters (15)
- 0:00Introduction to Supercloud 4
- 0:21First reaction to ChatGPT
- 1:45ChaosSearch role in AI
- 2:52Integrating LLMs with data
- 4:15Bring your own LLM approach
- 5:20LLM integration challenges
- 6:42Teaching prompt engineering
- 7:47Splunk acquisition impact
- 9:12Data prep and movement
- 10:41Vector databases perspective
- 12:12Natural language data interaction
- 13:34Future of data and AI
- 14:15Minimizing LLM hallucinations
- 16:01Customer AI challenges
- 17:04Closing remarks
