Swiss Army Django: Small Footprint ETL
summary
Case study of a single-app ETL that scrapes nested game data, enriches it, and exposes it via GraphQL and Discord; code uses async Django with background loops started in app ready callbacks, HTTP fetches with httpx, schema served by strawberry and simple cron scheduling via cronoter. Covers concrete trade-offs: no durable queue (vs Celery), async ORM limits (no async transactions, queries serialize), sharding and CPU offload strategies, and live-build static dashboards — system runs in one process.
AI-written summary · about
chapters (16)
- 0:00Small footprint ETL intro
- 1:25What ETL means
- 3:28Django and async for ETL
- 5:53Async Django setup
- 7:57Fault tolerance without Celery
- 9:42Async ORM limits
- 10:46Basic ETL example
- 12:55Scheduling ETL tasks
- 14:19GraphQL for ETL data
- 19:58GraphQL performance
- 21:21Extending ETL with chatbots
- 23:48Scaling ETL systems
- 25:11ETL in Kubernetes
- 26:35PEP 703 and Python’s future
- 28:03Legal web scraping
- 28:48Q&A
related talks
19mSwiss Army Django: Small Footprint ETL
47mBuilding Scalable Web Apps
38mButter smooth, interactive applications with Django...
28mBoosting Python Web Apps performance with Celery and Concurrency
54mAdvanced Aspects of the Django Ecosystem: Haystack, Celery & Fabric
41mEmbracing the Monolith in Small Teams
23mµDjango, an Asynchronous Microservices Technique with Practical Examples
41mBuilding high-performance, type-safe GraphQL APIs with Strawberry and Django