The Imitation Game: Encapsulate Your OSB Workload at Scale
summary
OpenSearch Benchmark now generates privacy-safe, production-like datasets and streams them into tests at terabyte scale. It offers synthetic data generation (index-mapping rules or custom Python modules), a Dask-powered parallel generation pipeline for fast bulk output, and streaming injection that pulls objects from S3 and feeds clustered clients to sustain multi‑TB/day ingestion. Demonstrated features: example configs, profiling of Python generators, on‑the‑fly chunking, and real-time Dash monitoring.
AI-written summary · about
chapters (18)
- 0:00Open Search Benchmarking intro
- 1:49Speakers and project overview
- 2:55OSB workload limitations
- 4:40Custom workloads in OSB
- 6:48Production workload pain points
- 7:30Synthetic data in OSB
- 10:00Realistic workload data
- 11:24Synthetic data approaches
- 13:53Demo: synthetic data generation
- 18:54Synthetic data recap
- 19:59Custom workloads at scale
- 21:27OSB 1.0 limitations
- 23:54Streaming injection in OSB 2.0
- 27:48How streaming injection works
- 30:39Demo: streaming injection
- 32:52Custom vs generalized workloads
- 33:56OSB applications and future
- 35:22Resources and community
related talks
40mPyData Stack: Pure Python open source data platforms
41mUnifying Diverse Logs in Big Data Systems for Seamless Analysis and Action With OpenSe...
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
16mCompute Engine Testing with Synthetic Data Generation
16mA Thrilling Journey into Data-Driven Security Automation
51mBuilding a Self-Service Marketplace for Test Data
01h 06mBest practices when accessing Big Data or any other data!
19mAn Experimentation and Analytics Framework for Large-Scale AI Operations Platforms