A pythonista’s pipeline for large scale geospatial analytics
summary
An automated pipeline maps billions of device locations daily to millions of POIs using HDFS storage, geospatial joins and a Pythonic stack. It favors geohashing over R‑trees for scale, hashes user IDs for uniform subsampling to build lookalike controls, and uses K‑means to bin long‑tailed origin distributions. Impala handles short fast queries but the system moves to Spark for large jobs; outputs land in a dashboard served by Flask and Celery web portal, delivered to production within months and actively used for visitation and co‑visitation measurement.
AI-written summary · about
chapters (19)
- 0:00Python geospatial pipeline intro
- 1:04Problem and solution overview
- 3:56Architecture and Python tooling
- 6:24Impala and Spark for computations
- 9:39Data flow to points of interest
- 11:26Location encoding with geohashing
- 13:14Creating lookalike control groups
- 15:23Measuring co-visitation patterns
- 16:26K-means clustering aggregates
- 17:54Distance travel metrics
- 19:20Summary
- 20:04Q&A: Ellipsoids and GPS resolution
- 21:28Q&A: Geospatial data scale
- 22:12Q&A: Impala vs Spark trade-offs
- 23:18Q&A: UI flexibility
- 24:47Q&A: Pokémon Go potential
- 25:11Q&A: Handling temporal data
- 26:38Q&A: Traditional vs cluster DBs
- 27:20Closing remarks
related talks
26mWho Said Wrangling Geospatial Data at Scale was Easy?
50mBig Data Processing and Data Science Environment in Python
20mCloud Native Patterns for Geospatial Analytics and AI
30mGeospatial analysis made easy with PostGIS and Geoalchemy
41mMassively Parallel Processing with Procedural Python
19mBig Data
01h 35mDeep Dive into the Big Data Landscape
35mData Processing Pipelines for Small Big Data