Pugs, Poe's and pipelines; An engineering perspective on big-data streams...
summary
Argues that early, local reduction of telemetry cuts latency and cost, introducing Point of Enrichment (PoE) and Point of Aggregation (Pug) as primitives. Uses Fastly examples: eBPF, sFlow and logs feed autopilot; runtime Pugs inside connection buffers drop memory by an order of magnitude; Kafka consumers and HTTP buffers are natural pug sites; ClickHouse SummingMergeTree provides persistent aggregation. Reports routine reductions like 500k rows/s -> 50% with a 15s aggregation window and says these approaches materially lower query time and spend.
AI-written summary · about
chapters (18)
- 0:00Introduction and welcome
- 1:43Problems with watershed metaphor
- 3:29Telemetry data as torrential flow
- 5:14Autopilot and data pipeline needs
- 6:58Pipeline cost and latency factors
- 9:07Budgeting data pipeline resources
- 10:12Meet Po and the Pug
- 12:36Point of enrichment explained
- 13:20Point of aggregation explained
- 15:47Free Pugs in existing workloads
- 17:56Windowed telemetry and Pugs
- 19:00Pugs in production at Fastly
- 20:07Resituating workloads around Kafka
- 21:53Multiple Pugs and pipeline design
- 23:39Building custom Pugs
- 25:03Pugs in persistence layers
- 26:49Pipeline design recommendations
- 27:38Q&A session
related talks
30mFive Challenges in Real-time Stream Processing and Five Solutions
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...
24mA Gentle Introduction to Building Data-Intensive Applications
39mAdvanced Stream Processing on the Edge
21mSplunk Data Stream Processor
47mBuilding Modern Data Streaming Apps with Open Source
16mLife is But a (Data) Stream: Building Quality Data Pipelines
39mCodeless Pipelines With Pulsar and Flink