Valuable LLM lessons learnt on Kaggle's ARC AGI Challenge
summary
Six months of work produces an automatic program generator that runs Llama3 locally via llama.cpp on an RTX3090, generates Python/NumPy solutions, executes them in isolated subprocesses (joblib) and scores correctness against Kaggle ARC AGI tasks. Raw JSON representation gives ~12% success, JSON+CSV gives 54% and repeated runs yield ~42% for one-pass; adding iterative feedback where the model sees which examples passed lifts performance to iterated feedback reaches 75%. Details: code generation frequency, runtime tradeoffs (3B vs 70B), and representation choices decide success.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:08Kaggle ARC AGI Challenge
- 7:02Local LLM setup & prompt engineering
- 10:09Program-aided language models
- 13:16Evaluating LLM solutions
- 15:21Data representation in prompts
- 18:29Iterative feedback & self-improvement
- 21:39Key takeaways & next steps
- 25:10Feature engineering in prompting
- 26:36Q&A
related talks
27mAI Demystified Unveiling the Intelligence Behind the Machine
18mLLMs and GenAI: A Guide to Building Your Own RAG Model for Document Interaction
55mLessons Learned From Implementing Generative AI Solutions
29mAI-Driven Code Generation and Testing: Accelerating Development with GenAI
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
55mBuild enterprise generative AI apps using Llama 3 at 1,000 tokens/s on the SambaNova AI platform
47mBuilding and Testing AI-Agent Powered LLM Applications: A Live Demonstration
43mAI Development Lifecycle: Learnings of What Changed with LLMs