Streaming Data Pipelines to Enhance Real-Time Decision-Making
summary
A live demo shows a booking agent failing when inventory is seconds out of date, then presents StreamSets (now part of IBM) as a no‑code streaming platform that claims 200 drag-and-drop stages and 90–95% no-code pipeline coverage, with optional Groovy/Java/JSON scripts for edge cases. Pipelines use schema-on-read to tolerate data drift, can swap targets (Snowflake or Databricks) without rewiring, run on‑prem/cloud/edge, and the vendor cites fraud loss multipliers and a cited $1M/company (~$1.1T total) cost from stale inventory.
AI-written summary · about
chapters (11)
- 0:00Real-time data importance
- 1:46Why real-time streaming matters
- 3:14Costs of delayed processing
- 5:27Challenges in stream processing
- 6:28StreamSets introduction
- 6:48StreamSets drag-and-drop builder
- 8:56Reliability and scalability in StreamSets
- 9:17Handling data drift
- 10:22StreamSets deployment options
- 11:05Key benefits of StreamSets
- 12:07StreamSets in IBM’s suite
related talks
20mReal-Time Data Integration
01h 20mSupercharge Your Real-Time Analytics: Snowpipe Streaming & Dynamic Tables in Snowflake
24mBreak Barriers In Real-time With Data Streaming
30mFive Challenges in Real-time Stream Processing and Five Solutions
42mApache Spark Declarative Pipelines in Action
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success
08mReal-Time Data Systems: Speed, Scale, and Resilience
16mLife is But a (Data) Stream: Building Quality Data Pipelines