Swimming in the stream...
summary
Live-coded pipeline uses Akka Streams to ingest a random request generator, enrich each tuple with customer/product lookups (fake ODB returning Futures), call an external market-price HTTP service via an HTTP connection pool flow, then emit enriched records with BigDecimal prices. Pipeline demonstrates backpressure, async Future handling without manual blocking, a broadcast that produces per-30-second substreams and computes total and average by product and by customer, and runs on a single machine; the demo mostly works with one small probe miswire fixed live.
AI-written summary · about
chapters (8)
- 0:00Introduction to streaming
- 3:24Streaming concepts and Akka
- 6:48Building the data flow
- 10:41Integrating external services
- 15:51Handling HTTP responses
- 19:22Writing results to file
- 20:31Real-time aggregations
- 22:47Summary and Q&A
related talks
48mTaming & Composing High Performance Stream-Oriented Processing Systems
14mStreaming Data Pipelines to Enhance Real-Time Decision-Making
30mFive Challenges in Real-time Stream Processing and Five Solutions
19mBuilding Stateful Streaming Pipelines
42mPrinciples and Patterns for Streaming Data Analysis
53mIntroduction to data streaming
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
42mBuilding Robust Streaming Data Pipelines with Apache Spark