Linguistic, domain and task adaptation of LLMs using the PLLuM models as an example
summary
The talk evaluates language, domain and task adaptation for PLLuM-style LLMs to prevent harmful cross-language interpolation. It reports that continued pretraining on Polish corpora, followed by domain-specific pretraining and supervised fine-tuning on multi-instruction data, produces usable outputs; experiments used a Plum 12B base, ~936 targeted instructions for word-sense clustering and ~100 exemplar instructions per official-text genre. Results show small models struggle with JSON formatting, while Deepsek/Deepsik V3 achieved nearly 100% parsable outputs with coherence ≈0.76 and completeness ≈0.79.
AI-written summary · about
chapters (13)
- 0:00PLLuM adaptation intro
- 1:48How LLMs work and limits
- 3:13LLM adaptation types
- 6:25Interpolation examples
- 8:13Language adaptation for emails
- 10:19Adapting to official texts
- 12:07Adapting to conversational analytics
- 13:54Word meaning clustering
- 17:31PLLuM fine-tuning experiments
- 21:44Fine-tuning insights
- 23:30PLLuM instruction sets
- 24:34Key insights summary
- 25:17Q&A
related talks
49mA Practical Approach to Using LLMs: Peculiarities and Pitfalls
46mUnder the Hood of LLMs — all those questions you never dared ask about them
29mLocal LLM fine-tuning: a practical example
21mFrom Data to Precision: Fine-Tuning LLMs for Proprietary Query Languages
20mKnowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs
42mKeynote: Foundation to Build LLMs
26mModern NLP for Proactive Harmful Content Moderation
48mBridging the data gap between LLMs and children