Supercharge Your Workers with Storm
summary
Backend for Skylight runs on Apache Storm and shows how to replace DB coordination with spouts, tuples, partitions and states to scale realtime jobs; implements EWMA trending counts, uses Trident batching and partitioning to colocate keys, and persists at commit to get close to exactly‑once despite Storm’s at‑least‑once model; recommends JVM/JRuby bindings over MRI, mentions Redis and Kafka for queues; cites benchmark claims of one million messages per second and warns of operational overhead (Zookeeper, Nimbus, cluster rebalances).
AI-written summary · about
chapters (31)
- 0:00Intro
- 0:17SUPERCHARGE YOUR WORKERS WITH STORM
- 1:04WHAT IS STORM?
- 2:44FAULT TOLERANT
- 3:16REALLY FAST
- 3:29LANGUAGE AGNOSTIC supposedly
- 4:19RUBY DSL: REDSTORM
- 4:45TWITTER TRENDING TOPICS
- 5:33EXPONENTIALLY WEIGHTED MOVING AVERAGES
- 8:19TWEETS FROM A GIVEN HASHTAG STOP
- 9:20IS IT WEB SCALE YET?
- 11:36ENTER STORM
- 11:46STREAMS / TUPLES
- 12:39SPOUT / STATE
- 14:29MORE TRANSFORMS
- 17:05DEFINE THE TOPOLOGY
- 18:37RUN IT
- 19:17WINNING
- 24:48EXECUTION IN BATCHES
- 29:48HOW MANY PARTITIONS?
- 30:09MANY SERVERS
- 30:19MANY THREADS PER SERVER
- 30:33MANY PARTITIONS PER THREAD
- 32:09#REALTALK
- 35:49MESSAGE PROCESSING GUARANTEES
- 36:22STORM IS AT LEAST ONCE
- 36:43MONOTONICALLY INCREASING BATCH IDS
- 39:49REQUIRES TRANSFORMS TO BE PURELY FUNCTIONAL
- 40:21REQUIRES THE SPOUT TO RE-EMIT IDENTICAL BATCHES
- 40:52TL;DR
- 41:07GOOD FOR COMPLEX DATA PROCESSING FLOWS
related talks
39mBuild blazingly fast, ultra lightweight cloud native Java enterprise applications
27mLeveraging Event Streaming to Super-Charge your Business Solutions.
47mBuilding High Performance Real-time Stream Processing Microservices
22mEmbracing Modern Workloads without Losing "IT"
43mIntro to Stateful Services
13mLow-Maintenance Backend Architectures for Scalable Applications
22mAntimony: a real time distributed stream processing system
40mUncoupling Systems