How to Implement Online Search Quality Evaluation with Kibana
summary
A reproducible Kibana pipeline that runs A/B comparisons of two rankers against live Elasticsearch logs, indexing interactions with fields for book_id, test_group, query_category, timestamp and binary impression/click/add_to_cart/sale flags, then computing metrics with bucket script CTR calculation and per-device filters. Dashboards show per-product, top-query and result-count slices (<=10, 11–50, >51) and use Vega query-frequency buckets for custom plots; an intersection of query ID sets implements a common-query intersection filter. Implementation exposes frontend-data bugs and successfully surfaces model wins and regressions.
AI-written summary · about
chapters (9)
- 0:00Introduction by speakers
- 2:12Overview of talk agenda
- 2:55Online search quality evaluation basics
- 7:13A/B testing fundamentals
- 10:33Kibana implementation steps
- 14:32Time series visualization with Kibana
- 21:32Advanced Vega visualizations
- 29:49Key takeaways summary
- 34:27Q&A session
related talks
15mFrom Top N Lists to Visual Analytics: Query Insights Dashboard in OpenSearch
20mMeasuring Data with OpenSearch to Better Serve the Community
34mMetrics driven development
17mMastering the Query Insights Dashboard in OpenSearch
19mUser-centered observability: load testing, real user monitoring & synthetics
39mTransform data investigation via Elasticsearch Query Language (ES|QL)
33mAnalytics Data Driven Testing
44mIterative metrics, dashboards and monitors