Riak Pipe, Distributed Data Processing

on YouTube ↗45m

" summary

Riak Pipe is a Basho-built pipeline layer on Riak Core that runs Erlang or JavaScript fittings on the cluster’s Vnodes to do MapReduce-style work where the function moves to the data. It supports pre-reduce to cut network shuffle, uses Vnode-local queues and worker pools for backpressure, and is at-most-once processing. It’s battle-tested in hundred-node clusters, used for Riak MapReduce and S3-style billing, and exposes plans for external language hooks.

AI-written summary · about

" chapters (13)

1. 0:00Introduction to React Pipe
2. 1:36React Core foundation
3. 6:25Dynamo system architecture
4. 13:29Working with pipes and fittings
5. 20:06Distributed processing patterns
6. 23:26Event driven design
7. 25:15MapReduce implementation
8. 28:41Cloud storage use case
9. 30:31Current users and stability
10. 32:01React Pipe versus Storm
11. 34:13Future developments
12. 36:45Erlang ecosystem tools
13. 39:09Q&A session

" related talks

• 46mEvents Analysis with Riak, Perl and Erlang at Booking.comDamien Krotkine · Code Mesh 2015
• 44mRiak, Redis, Apache Solr & Spark: Deploying with the Basho Data PlatformBryan Hunt · GOTO London 2015
• 31mRiak's RebirthNicholas Adams · Code BEAM SF 2019
• 26mData Processing on RaySangBin Cho · Ray Summit 2021
• 04mJan Plosker at RIT++ 2012Yan Plosker · РИТ++ 2012
• 38mRTB and Big Data - where Erlang and Hadoop meetDennis Meyer, Zubair Sheikh · Berlin Erlang Factory Lite 2015
• 39mAn adventure in distributed programmingWiebe Marten Wijnja · ElixirConf EU 2019
• 41mBreakout SessionDerek Nelson · PGConf Silicon Valley 2015