Summingbird: A Streaming Map-Reduce API for Storm, Hadoop, & More
summary
Summingbird is a Scala library that expresses streaming map-reduce as composable producers and planners, running on Storm, Hadoop and planned Spark backends; it encodes reductions as semigroups/monoids and ships a large algebra library (algebird) plus a store combinator layer (Storehouse). Implementations back existing Twitter pipelines processing billions of events a day, use sketching (HyperLogLog, Count-Min) for approximate counts, emit platform-specific plans, and support stores like Redis, Memcached, MySQL and Cassandra while sharing graph optimizations.
AI-written summary · about
chapters (10)
- 0:00Intro
- 1:09What was the problem
- 3:23Why use Storm
- 4:55The problem domain
- 8:19The stores
- 9:28The signatures
- 11:25Mono aids
- 15:34Producer
- 17:34Reductions
- 23:59Store implementations
related talks
42m"Distributed stream processing in Ruby"
39mReal Time Datamining and Aggregation at Scale
31mHelp I need a stream processor - learning how to chose ...
42mApache Beam: A Unified Model for...
34mBig Data, Small Code ... #bbuzz
37mStorm, a Distributed Real-time Data Processing System
25mMergeable Summaries for High Scale, High Cardinality Analytics
21mApache Beam (incubating)