The (mis)use of GenAI in Data Analytics
summary
Generative models often get misapplied to forecasting and regression where simple ETS or ARIMA beat transformers, and using LLMs for time series or sales numbers leads to dangerous hallucinations; tokenization differences make dates and numeric parsing brittle across models, and frontier math benchmarks drop from ~80–90% to single digits on private problems (sometimes ~2%). Retrieval Augmented Generation relies on retrieval not generation — BM25 outperforms embeddings in my tests — and you can often use Postgres vector search; evaluate from day one with logged prompts and binary metrics.
AI-written summary · about
chapters (18)
- 0:00GenAI misuse in data analytics
- 0:46When not to use GenAI
- 2:12LLMs fail at forecasting
- 3:39LLMs vs classic ML
- 5:05Clean data’s critical role
- 6:27Tokenization pitfalls
- 8:58LLMs struggle with math
- 10:03Mitigating hallucinations
- 11:50RAG limits and basics
- 13:40Vector DB necessity
- 14:44Evaluation from day one
- 16:31Simple GenAI metrics
- 17:37Evaluation drift risks
- 18:42Clear accuracy goals
- 20:08LLMs lack reasoning
- 21:34Agent-based analytics challenges
- 23:19Why analysts and data scientists aren’t obsolete yet
- 23:41GenAI strengths and limits
related talks
22mGenerative AI - A Pragmatic Business View
44mAI Engineering Anti-patterns
49mGenerative AI: Concepts, Tools, and Applications for Data Professionals
55mAI Showdown: The Good, The Glitchy, and The Risky
15mHarnessing the Power of Generative AI for Intelligent Applications
41mGenerative AI: bliss or miss?
17mUnlock the power of Generative AI: From understanding its principles to utilizing its power
41mThe Unreliable Computer Revolution - Embracing the Double-Edged Sword of GenAI