Reporting Systems Built on Top of Hadoop & Lucene
summary
Zanox joins tracking (2M sales, 30M clicks, ~1B views daily) with search-cost feeds using Hadoop, HDFS and MapReduce to produce Lucene indices that power reporting for millions of publishers and thousands of advertisers. Daily ingest is ~0.5 TB, batch processing takes ~6 hours, index artifacts of ~300–400 GB deploy in ~28–30 minutes, and online queries hit sub-second responses via a topology of mergers, searcher groups and an observer. A custom partitioning approach yields up to 4x speedups (typical ~30%) and the stack moves from proprietary orchestration toward Oozie integration.
AI-written summary · about
chapters (4)
- 0:00Introduction to reporting
- 3:42Reporting system architecture
- 11:14Joining datasets efficiently
- 17:29Q&A session
related talks
30mAn architecture for big data: Supporting journalism through transparency in advertising
53mBig Data: from mammoth to elephant
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
28mA complete Tweet index on Apache Lucene #bbuzz
01h 04mOnline Advertising: Hybrid Approach to Large-Scale Data Analysis
45mReal Time Analytics with MongoDB
29mThe Data Journey
01h 06mYahoo!, Big Data, and Microsoft BI: Bigger and Better Together