How to handle the luxury of having too much training data
summary
Claim: too much labeled aerial data can hurt models unless selection is controlled. Built an in-memory streaming rig in Python with multiprocessing and shared queues, using an SQLite building-area buffer to prefilter tiles and in-memory sampling with 40 parallel generators and reuse up to 10×. Trains at the GPU rate (24 images/s) without terabytes of precomputed tiles, improves rural true detections from 77% to 88% and cuts false positives (300→187), and feeds results into a municipality property‑control workflow.
AI-written summary · about
chapters (9)
- 0:00Introduction to AI at Nodecraft
- 2:20Building detection in Norway
- 6:18Data sourcing and labeling
- 10:33Challenges with static data
- 14:28Initial model performance
- 20:55Optimizing the training rig
- 34:42Performance improvements
- 36:45Demo and workflow demonstration
- 43:02Q&A session
related talks
28mData Curation: Transforming Bytes into AI Gold
18mUnleash your Private Datasets to Power up AI
24mTiling & Parallel Processing of Large Images
01h 36mFrom Maps to Models: An End-to-End Journey in Geospatial Machine Learning with Python
34mDeep Learning at Scale
21mNeedle in the Haystack: Applied ML for Inspections of Power Infrastructure
19mAccelerating ML Development with Multi-Modal Datasets: Python, Parquets, and Daft
31mReliable Data for Large ML Models: Principles and Practices