'Write a Scraper as a State Machine with Playwright and Kafka'
summary
A Glassdoor scraper written in TypeScript runs on Node.js and uses Playwright for browser automation, XState to model the scraper as a state machine with guarded transitions, parallel states for concurrent profile scraping and resume downloads, and Kafka to decouple job producers and success/failure topics. It logs in with provided credentials, navigates pages, waits for network events, and saves downloadable resumes as Node readable streams (download can return null). Machine logic is unit and integration tested; promise-based invokes update context and publish results to Kafka.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:29Establishing expectations
- 1:34Playwright vs Puppeteer
- 7:17What is XState?
- 9:23XState: Example (1/2)
- 10:34XState: States
- 11:21XState: Events
- 11:50XState: Context
- 12:50XState: Actions
- 15:47Kafka: Example architecture
- 16:58What is Apache Kafka?
- 20:48Takeaways
- 22:26Additional material
related talks
28mRethinking Browser Automation: Unleashing the Power of Puppeteer and Playwright
43mReal World Playwright Case Studies That Will Change Your Perspective!
42mMichal Slezak: Creating a Framework with Playwright - How to Do It Right?
46mWhy does building a robust automation framework matter?
25mPlaywright Test Runner
01h 31mAdvanced Playwright with AI
41mTesting Web Applications with Playwright
51mAdvanced Playwright Techniques for Flawless Testing