Integrating the Gemma model family in Transformers
summary
Finished integrating Gemma 3 in Transformers and shipping device‑agnostic code with integration tests, CLI support and Hugging Face repo weights for reproducibility. Key practical notes: attention masks for mixed image and text tokens require a sliding‑window causal mix, provide token IDs and sample inputs, and include reproduction snippets. Watch out for a bfloat16 position‑ID bug that breaks RoPE when position IDs get cast, and for alternate RoPE formulations that force rotary embedding weight rotations and careful head resizing to avoid corrupting token distributions.
AI-written summary · about
chapters (6)
- 0:00Why integrate with transformers
- 3:41Best practices for integration
- 7:27Collaboration and tools
- 9:33Integration workflows
- 10:52Debugging common issues
- 12:45Advanced model challenges
related talks
10mDemo: Using Gemma with the Hugging Face ecosystem
39mFlutter Project with Gemma: A State-of-the-Art Open Model
12mGemma: The responsible way to build
16mOpen models in the Gemini era
10mAI for everyone with Gemma
44mBuilding amazing Front-End Project with Gemma: A State-of-the-Art Open Model
08mAI-Powered transformation: Driving innovation and reshaping organizations
15mGemma, DeepMind's Family of Open Models