Ingestion at Scale: Open Source Pipelines for Getting Data Into Open...
summary
Shows concrete pipelines using Fluent Bit, Data Prepper, OpenTelemetry Collector and Kafka to feed OpenSearch with parquet output to S3 now supported in Fluent Bit (merged Monday), plus Data Prepper pulling parquet and writing to OpenSearch. Recommends transform at the edge to save server cost, use object store or Kafka as durable buffers for hundreds of thousands of clients and hundreds of terabytes/day, enforce dead letter queue and consistent bulk sizes, and monitor Prometheus/OpenTelemetry metrics to detect backpressure.
AI-written summary · about
chapters (13)
- 0:00Scalable log ingestion
- 3:28Ingestion pipeline anatomy
- 7:25Collectors for pipelines
- 9:12Fluent Bit high-scale collector
- 13:07Vector and OpenTelemetry collectors
- 15:14Kafka in high-scale ingestion
- 16:38OpenSearch Data Prepper
- 18:44Scalable ingestion considerations
- 23:22Object storage as log buffer
- 27:12Kafka and object storage patterns
- 28:15Parquet and direct query patterns
- 34:55Summary and future directions
- 35:15Q&A on overflow
related talks
31mFluent Bit & OpenSearch: The Power Couple of Open Source Observability
26mUsing Data Prepper for Observability Ingest
25mPersistent and performant ingestion using Data Prepper
47mBuilding Modern Data Streaming Apps with Open Source
25mRunning OpenSearch in the Real World: Migration, Mistakes and Maturity
43mBuilding Scalable and Extendable Data Pipeline
34mBuilding and Scaling Robust Zero-code IoT Streaming Data Pipelines
44mProcess and analyze millions of events per second with open source tools