Teaching Exotic Programming Languages to Large Language Models
summary
Claims that models taught with continuous pre-training on whatever code and docs exist plus an instruction-capability transfer via fine-tuning reuse can actually write nonstandard languages instead of hallucinating, tested with retrieval-augmented generation, few‑shot prompting and instruction fine‑tuning. Few hundred to few thousand GitHub repos for Gleam and Moonbit yield near‑0% baseline pass rates on translated HumanEval and a harder maze benchmark, RAG/few‑shot fix trivial problems but the continuous pretrain + reuse pipeline produces the largest, measurable boost though still short of Python/Java quality.
AI-written summary · about
chapters (11)
- 0:00Teaching exotic languages to LLMs
- 2:25LLM limitations with exotic languages
- 5:14How LLMs learn code
- 11:20Data collection challenges
- 13:49Evaluating code generation
- 16:18In-context learning techniques
- 21:02Training for language adaptation
- 23:50Limitations of pre-training & tuning
- 26:00Fine-tuning reuse
- 28:58Key takeaways & future work
- 30:08Q&A
related talks
42mPost-Training Small Language Models: the adventures of a practitioner
01h 38mTraining Large Language Models: Practices and Research Questions
01h 02mAI Effects on Research and Education: A Programming and Software Language Perspective
15mKeynote: Pre-training and Fine-tuning of Code Generation Models
20mKnowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs
22mSpecializing Small Language Models with less data
40mIs my Large Language Model a Strange Loop?
27mBuilding Mini Minds Our Adventures in Teaching Small LLMs to Reason