Real Time Data Warehousing: A Journey from Batch to Streaming with Faust
summary
CERN moves a recruitment ETL from 15‑minute cron batches to a Faust-based streaming pipeline that stops lost intermediate statuses by ingesting the platform webhook to an HTTP endpoint and immediately producing into Kafka. Faust app with an app.page that receives webhooks, publishes to a rec_events topic, and lightweight deployment on Kubernetes via Git/Argo CD agents consume the generic topic, route messages into per-entity topics (candidate, job, application) and write history rows into Oracle The Python/low‑volume choice works in production.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:07Data warehouse challenges
- 3:39Batch processing problems
- 5:26Streaming ETL solutions
- 6:52Why Faust
- 7:14Faust and Kafka
- 8:19HTTP endpoints with Faust
- 9:21Faust agents
- 10:23Streaming pipeline
- 11:05Deployment with Argo CD
- 12:11Q&A
related talks
28mBuilding Event Streaming Applications with Serverless Kafka
39mDeploying Fast Data Pipelines
30mFive Challenges in Real-time Stream Processing and Five Solutions
35mStreaming and Real-Time Analytics
43mA Data Streaming Architecture with Apache Flink #bbuzz
46mKafka Streams at the Big Data Crossroads at Lightning Speed
30mIntroduction to Data Streaming
42mPrinciples and Patterns for Streaming Data Analysis