Processing Billions of Events in Real-Time with Heron

on YouTube ↗50m

" summary

Processes tens of billions of events per hour using an open‑source engine called Heron, running in production for 2+ years on a multi‑tenant Mesos/Aurora cluster equivalent to a few thousand machines. Architecture uses a topology master plus stream managers and process‑per‑task for debugability and heap dumps, supports at‑least‑once and at‑most‑once semantics, and enforces cluster fairness via a distributed back‑pressure protocol that slows spouts. Twitter reports up to a 10x CPU reduction versus Storm; Python and C++ instances and scheduler abstraction are already implemented.

AI-written summary · about

" chapters (38)

1. 0:00Intro
2. 0:31TALK OUTUNE
3. 1:17TWITTER IS REAL TIME
4. 3:27HERON TERMINOLOGY
5. 4:39HERON TOPOLOGY
6. 4:56WORD COUNT TOPOLOGY
7. 7:16STREAM GROUPINGS
8. 8:37WHY HERON
9. 10:32HERON DESIGN DECISIONS
10. 12:00HERON ARCHITECTURE
11. 13:10TOPOLOGY ARCHITECTURE
12. 18:23TOPOLOGY EXECUTION
13. 19:17TOPOLOGY MASTER
14. 19:49STREAM MANAGER
15. 20:40HERON INSTANCE
16. 22:00HERON DEPLOYMENT
17. 22:47HERON SAMPLE TOPOLOGIES
18. 23:30SAMPLE TOPOLOGY DASHBOARD
19. 23:56HERON TWITTER
20. 24:30HERON USE CASES
21. 26:32HETEROGENOUS ECOSYSTEM
22. 27:47LOOKING BACK-MICROKERNELS
23. 28:17HERON-MICRO STREAMING
24. 31:51HERON - MICROSTREAM ENGINE
25. 32:38HERON ENVIRONMENT
26. 34:07APPROACHES TO HANDLE STRAGGLERS
27. 34:44DROP DATA STRATEGY
28. 35:15SLOW DOWN SENDERS STRATEGY
29. 38:15BACK PRESSUREN PRACTICE
30. 40:24DETECTING BAD FAULTY HOSTS
31. 41:21HERON RESOURCE USAGE
32. 43:04PROFILING SPOUTS
33. 43:48RESOURCE CONSUMPTION - BREAKDOWN
34. 44:16HERON EXPERIMENT
35. 44:34HERON PERFORMANCE
36. 45:48INTERESTED IN HERON?
37. 46:00HERON CONTRIBUTIONS
38. 46:21HERON RESEARCH PROBLEMS

" related talks

• 52mProcessing Billions of Events in Real-Time with HeronKarthik Ramasamy · KubeCon + CloudNativeCon North America 2016
• 39mTwitter Heron in PracticeKarthik Ramasamy · Data By the Bay 2016
• 36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzzAmeya Kanitkar · Berlin Buzzwords 2015
• 37mStorm, a Distributed Real-time Data Processing SystemMariusz Gil · Confitura 2013
• 46mClojure Scaling the Event StreamDerek Troy West · Clojure/conj 2017
• 37mReactive Architecture on AWS: from 0 to 1.6M events/minuteRafael Ramos, Juraj Zachar, Chase Abendschein · AWS Summit Berlin 2023
• 01h 11mAsychronous design with Spring and RTI: 1M events per secondStuart Williams · SpringOne2GX 2014
• 20mBuilding Apache HeronJosh Fischer · BazelCon 2018