From Treatise to Tech: The Development of ChatSOC, an AI Research Assistant
summary
Sullivan on Comp builds ChatSOC, an AI legal research assistant that starts by failing with self‑hosted Llama/GPT-style models, then switches to retrieval-augmented generation via the OpenAI API and achieves usable results. A two-person dev team plus content experts creates a benchmarking loop using a 600-question gold set to score iterations, lifts accuracy well above the initial ~50% baseline, and ships the feature inside subscriptions to ~8,000 customers; costs track to chats but fall after migrating to GPT‑4o.
AI-written summary · about
chapters (9)
- 0:00Introduction and setup
- 1:08From legal treatise to AI assistant
- 3:12Inspiration and early experiments
- 5:16Switching to RAG and progress
- 7:48Testing and refining accuracy
- 9:56Building a benchmark dataset
- 12:42Key lessons learned
- 14:30Cost management and scaling
- 15:37Q&A session
related talks
48mMastering the AI Shift: Adaptive Strategies for Software Evolution
27mBuilding AI applications: Successes and Lessons
16mUnveiling the practical prowess of OpenAI's ChatGPT
32mChatBot Learning Assistant with Large Language Models
01h 09mChatGPT and OpenAI API solutions: 👑 My Favourite Implementations
46mUsing AI in Software Design: How ChatGPT Can Help With Creating a Solution Arch.
51mChatGPT as a developer's personal servant
39mBuilding Your Own Bot with GPT: Harnessing the Power of AI...