Persistent and performant ingestion using Data Prepper
summary
Data Prepper provides durable, pluggable ingestion: an HTTP push path can persist incoming events to a Kafka buffer which responds 200 once data lands in Kafka, then consumes and commits only after a successful write to OpenSearch. Pull sources (S3 via SQS, Kafka) use end-to-end acknowledgements with an easy acknowledgements=true flag and a visibility duplication protection option. Failures go to an S3 or file dead-letter queue that records errors for replay. Default bulk target is 5 MB; tests recommend ~20 MB for better throughput and observed higher performance than Logstash.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:04What is Data Prepper
- 6:32Data Prepper buffering
- 10:43Handling failure situations
- 15:19Improving performance
- 16:41Wrap up
related talks
26mUsing Data Prepper for Observability Ingest
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...
48mContinuous Optimization for distributed BigData analysis
27mUsing Akka Streams for Web scale Data Ingestion
42mBuilding a Data Pipeline with Clojure and Kafka
43mDeep Dive Into OpenSearch Pull-Based Ingestion at Uber
34mBuilding and Scaling Robust Zero-code IoT Streaming Data Pipelines
43mBuilding Scalable and Extendable Data Pipeline