Who needs ChatGPT? Rock solid AI pipelines with Hugging Face and Kedro
summary
Shows a concrete pipeline that extracts ~200 Mastodon posts into Minio/Delta, cleans text with Polars, loads Hugging Face models (a ~1GB summarizer and a fill‑mask model) as Kedro catalog datasets, generates hashtags and a short summary, and writes an executive_summary.txt back to S3 — everything runs end‑to‑end. Surprising details: ~400,000 HF models exist, model weights download cost and memory matter, and Kedro pipelines keep IO declarative while Hugging Face supplies the checkpoints.
AI-written summary · about
chapters (8)
- 0:00Open vs commercial AI models
- 7:33Kedro for robust AI pipelines
- 16:23Kedro architecture overview
- 21:54Demo setup and Mastodon data
- 39:03Hugging Face with Kedro
- 51:19Reporting pipeline step-by-step
- 1:08:00Live coding final pipeline
- 1:18:11Conclusions and future
related talks
55mCloning ChatGPT with Hugging Face and Elasticsearch
41mCloning ChatGPT with Hugging Face and Elasticsearch
31mHugging Face ecosystem for Local AI/ ML
19mBuilding Machine Learning Applications using Hugging Face
01h 03mIt's All About AI
26mOne Pipeline to Rule Them All: Unifying Multimodal and AI Data Processing with Daft
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case S...
27mGen-AI at Scale: Simplifying Orchestration of Healthcare Applications Across Multi-Cluster Environme