Scraping the Dungeons. Find your perfect D&D game using Scrapy and Bokeh visualisation

on YouTube ↗29m

" summary

Builds a Scrapy pipeline that hits the site’s asynchronous GraphQL pagination, requests each game’s detail page to merge missing fields, and writes items to JSON; demo runs with 100 collected games (project setting closes at 1,000 elements) and produces a small Bokeh visualization with filters for tactical level and seats-left so you can spot likely-to-start games. Scraping works in the demo; handling IP blocking and production proxies is noted as costly and nontrivial.

AI-written summary · about

" chapters (12)

1. 0:00Introduction to D&D scraping
2. 2:10What is Dungeons and Dragons
3. 5:25Challenges of scheduling games
4. 7:56Project flow overview
5. 11:09Scrapy framework basics
6. 12:58Configuring Scrapy settings
7. 15:04Live site investigation
8. 18:20Building the spider
9. 22:36Running the spider
10. 24:01Visualizing with Bokeh
11. 25:44Real-world impact and tools
12. 26:07Q&A session

" related talks

• 29mSwiss Army Django: Small Footprint ETLNoah Kantrowitz · PyCon AU 2023
• 19mSwiss Army Django: Small Footprint ETLNoah Kantrowitz · DjangoCon US 2023
• 51mDungeons, Dragons, and Graph Databases.Guy Royse · Devoxx Ukraine 2020
• 02h 14mData Applications with BokehBryan Van de Ven · PyData San Francisco 2016
• 39mCreate Your Own Epic Adventure with PowerApps and Power BI: Dungeon and Dragons Edition!James Solan, Lee John, Matthew Bean · SQLBits 2023
• 40mOpen Data Dashboards & Python Web ScrapingMarie Whittaker · PyData DC 2016
• 59mBig data beautiful visualization on the browser with BokehFabio Pliger · EuroPython 2015
• 23mDjango's Data Science Makeover: Integrating D3.js and Bokeh for Data VisualizationDrishti Jain · DjangoCon US 2023