Swiss Army Django: Small Footprint ETL
summary
Single-app ETL that scrapes and enriches a game’s deeply nested data (items monsters quests), runs on one box, and serves results via Strawberry GraphQL and Discord. It uses async Django with uvicorn, HTTP extractors via httpx, background loops spawned with create_task, lightweight scheduling with chonor, incremental loads and Postgres history triggers, avoids Celery/RabbitMQ, and proves practical in production for frequent (seconds-to-hours) refreshes.
AI-written summary · about
chapters (10)
- 0:00ETL and side projects
- 1:26What is ETL
- 4:19Async Django for ETL
- 7:13Background tasks without Celery
- 10:45Async ORM limits
- 12:50HTTP clients for extractors
- 13:32Foreign keys in ETL
- 15:19Scheduling with Chon
- 16:22ETL history tracking
- 17:26Querying ETL with GraphQL
related talks
29mSwiss Army Django: Small Footprint ETL
29mScraping the Dungeons. Find your perfect D&D game using Scrapy and Bokeh visualisation
54mAdvanced Aspects of the Django Ecosystem: Haystack, Celery & Fabric
31mDjango for life (sciences)
23mThree Django Apps in a Trenchcoat: Writing an event management website with not a lot of time
53mDjango: a Data Shovel With a Future
28mOn a shoe-string and a t2.small: scaling on a [zero] budget.
25mFighting Homelessness with Django with Benjamin "Zags" Zagorsky