Keynote: The Evolution of Data Infrastructure at Splunk
summary
Splunk moves core processing from batch to an ingest stream by adding a firehose pub/sub and a Flink powered DSP that runs pipelines (SPL or Java SDK) for parsing, enrichment, joins, masking and routing. Scheduled polling workloads drop dramatically—examples show up to a 94% query reduction—and pipelines support denormalization, data quality fixes, role based masking, DR routing and autoscaled Flink clusters. The system ships curated streams, a dynamic function registry and early stateful functions support so stream processing replaces many view maintenance and alerting batches.
AI-written summary · about
chapters (18)
- 0:00Intro
- 2:02About Splunk
- 4:28Scheduled Queries
- 6:26Simple Architecture
- 8:22Latency
- 12:28Adhoc Queries
- 18:35Stream Processing
- 18:49Ingest Process
- 19:37Architecture
- 21:10Stream Processing Engine
- 23:23Firehose
- 28:59Integration
- 29:46Migration
- 32:15Functions
- 33:02Programming Model Changes
- 33:23RealTime Query
- 34:52Working with data
- 37:16Outro
related talks
21mSplunk Data Stream Processor
01h 33mSplunk as a Big Data Platform for Developers
27mRealise the value of data with Splunk
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
24mHow To Remove Data Siloes: Tips & Considerations
43mA Data Streaming Architecture with Apache Flink #bbuzz
36mSources, Sinks, and Operators: A Performance Deep Dive
28mBest of Stream + Batch: When Streaming Analytics Isn’t