Data Analysis and Map-Reduce with mongoDB and pymongo
summary
Live demos show mongoDB aggregation pipelines doing match project group unwind and date operators on a 37 GB dataset compressed to about 9 GB with 1.3 million playlists and ~17 million songs, driven from pymongo. Specific pitfalls include aggregation returning a cursor that is exhausted after list(), Python dicts losing sort order unless ordered, and BSON document and pipeline limits (16 MB result, ~100 stages). MapReduce remains useful when embedding JavaScript is required. iPython notebook and sample data are on the linked Bitbucket.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:31MongoDB
- 1:25MongoDB aggregation framework
- 2:13mongoDB pipeline
- 3:05pymongo
- 4:14demo structure
- 5:04Taylor Swift
- 5:37Build the first pipeline
- 11:58unwind operator
- 16:32stages
- 17:46grouping
- 20:15dates
- 23:14Match operator
- 23:32Competition
- 24:08unwind
- 26:56arm
- 29:10map operator
- 30:36MapReduce
- 34:50Best practices
- 37:16Useful resources
related talks
40mData Wrangling With MongoDB
45mReal Time Analytics with MongoDB
29mSwitching to MongoDB: The challenges of translating database schemas
59mCommon Query Patterns in MQL (Mongo Query Language)
37mDenormalizing Rails
30mHelp! I Need To UnSQLize My Application
48mGetting Non-Relational with MongoDB
43mHelp! I Need To UnSQLize My Application