Teaching and LLM to Catch British Sarcasm
summary
11,000 landlord interactions from ~2,000 accounts feed a production pipeline that uses Vox transcription, Watson transcription for calls, BigQuery for streaming and a Claude Sonnet LLM in a two-step LLM pipeline costing about a third of a cent per message. About 7% of messages hit the expensive structured extractor, ~1,100 calls were transcribed, and 36% of negatives are communication failures. Prompts plus ruley labels cut false red alerts and surface channel switches that predict preventable churn so the team builds routing and phone coverage fixes.
AI-written summary · about
chapters (12)
- 0:00Intro and Goal
- 1:15Sarcasm Problem Example
- 2:11Why Retention Matters
- 3:16Polite Dissatisfaction Signals
- 6:15Dataset and Channels
- 7:20Labeling Framework
- 8:35Two Step LLM Pipeline
- 10:04Prompt Engineering Fixes
- 11:54Production Deployment
- 12:13Key Findings and Metrics
- 14:03From Detection to Action
- 15:02Lessons and Wrap Up
related talks
32mDocuments Meet LLMs: Tales from the Trenches
44mProduction-Grade LLMs & AI Agents
33mLLM Telemetry as a First-Class Rails Concern
40mKeynote
50mKa-Ching! How Enterprises Measure LLM Performance and Cost
24mLLMs: Powering Lightning Fast Product Innovation
21mRevolutionizing Customer Care: Our Journey with LLM and RAG into Production
15mFrom Words to Data: Using LLMs to Turn Questions into SQL