Big Data, Small Computers
summary
Builds a distributed key-value store that uses UDP for control and TCP for data transfers, caches keys locally, and integrates HDFS pointers to handle terabyte-to-petabyte files. Implements a distributed fork/join runtime that checkpoints logarithmic-sized tasks so others can pick up work when nodes fail, and targets automatic parallelization of outer loops. Aims for JMM semantics (stronger than eventual consistency) with replication/repair, and shows a working prototype scaled to about 20 nodes.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:53MapReduce on Hadoop
- 1:32The Market
- 2:16Data Centers
- 2:48RAID Array
- 3:31Distributed Hashmap
- 5:15Memory Model
- 7:38Reliable Computation
- 9:02Fork Join
- 10:00Fork Joint
- 11:11Task Split
- 11:25Task Death
- 11:52Moron
- 15:59REST API
- 16:31Parallel Java
- 17:25HDFS
- 18:25Distributed KBstore
- 19:02Distributed forkjoin
- 19:26Programming model
- 20:21Appendonly
related talks
51mHigh Bandwidth Data & Low-latency Java
46mFrameworks Unplugged: Building Distributed Systems in Pure Java
52mDistributed Object Graphs with MicroStream
45mJava is the “big” in BigData
34mBig Data, Small Code ... #bbuzz
15mFaster Data Access with MicroStream
20mZero-Change Object Transmission for Distributed Big Data Analytics
01h 12mHidden pearls for high-performance-persistence in Java