Finding Order in Chaos, or MapReduce in the Age of the Internet
summary
Describes using MapReduce-style pipelines to process web-scale logs: Map extracts query strings, shuffle groups by key, Reduce counts frequencies and emits the top 1,000 queries from terabytes of logs. Discusses framework features that hide failures (retries, skipping bad records), cluster scheduling and monitoring, mentions Google MapReduce, Hadoop and Apache Crunch as runnable options, and recommends this batch approach once daily data reaches tens of gigabytes.
AI-written summary · about
