How Well do LLMs Detect Anomalies in Your Data?
summary
Baseline tests use OpenAI, BigQuery and Mistral and show concrete numbers: GPT‑4 detects about 32% → 68% of intended anomalies after prompt engineering, GPT‑3.5 ~16% on numeric data while GPT‑4 hits ~68% on numeric and ~100% on text. Prompt tricks—chain‑of‑thought (+8%), few‑shot (+204%), self‑reflection (+28%)—produce the big jumps but don’t sum to perfection. BigQuery’s anomaly model finds all intended anomalies but initially raises 21 false positives on 28 rows, which tuning and a 0.99999 threshold cut down to one.
AI-written summary · about
chapters (30)
- 0:00Introduction
- 2:10Data quality importance
- 4:37Traditional anomaly tools
- 5:18LLMs for anomaly detection
- 7:02Test setup electricity data
- 9:28Baseline GPT-4 vs GPT-3.5
- 9:48Chain of Thought prompting
- 11:55Few-shot learning boost
- 13:41Self-reflection multi-step
- 16:04Limitations of LLMs with numerical data
- 17:52Textual data anomalies
- 18:35BigQuery anomaly detection
- 21:45Fine-tuning BigQuery model
- 24:58Optimizing model terms
- 25:41OpenAI vs Mistral
- 26:23Summary
- 26:43Q&A: Percentages test cases
- 28:08Q&A: Feasibility of using LLMs for large-scale projects
- 29:11Q&A: Detection frequency
- 30:40Q&A: Explainability recall
- 32:07Q&A: Self-reflection false positives
- 32:48Q&A: Missing data handling
- 33:32Q&A: Unstructured data
- 34:15Q&A: Bank transaction anomalies
- 36:20Q&A: False positive rates
- 37:01Q&A: Numerical structural data
- 37:22Q&A: Next steps after detection
- 39:06Q&A: BigQuery unsupervised
- 39:26Q&A: ML models for numbers
- 40:31Closing remarks
related talks
24mHarnessing the Power of LLMs in Advanced Analytics
29mLies, damned lies and large language models
01h 04mCan you trust your (large language) model?
54mKeynote: Can you trust your (large language) model?
01h 05mCan you trust your (large language) model?
17mPsychology of AI
40mJudgement Day: Benchmarking "Black Box" LLMs With Open Legal Datasets
44mUnderstanding the abilities of AI systems: Memorization, generalization, and points in between