Ruby for the soul of Big Data nerds
summary
Company handles ~50 million monthly views and ~2 PB of video data, and builds a Ruby-based pipeline that collects 20x per-page events, filters with Fluentd, stores raw logs on HDFS and runs batch analysis with Hadoop/Hive and real-time joins with Storm via RedStorm on JRuby. A prototype clusters client bandwidths, reduces points for a globe dashboard, and feeds a fast rails-backed UI; experiments with GPU parallelism are ongoing.
AI-written summary · about
chapters (9)
- 0:00Start
- 5:36BigData & Its Challenges
- 8:37Evolving Trends
- 10:22Common Engineering Challenges
- 14:17Data Collection + Filtering/ Segmentation
- 19:12Storage
- 20:39Analysis
- 23:15Visualization
- 24:28STORM terminology
