800 mio. IP addresses to coordinates using Kafka Streams #bbuzz
summary
A Kafka Streams pipeline enriches roughly 800 million events/day by batching IP-to-coordinate HTTP calls with the Processor API transformer and punctuate timer, issuing 250-request bulk lookups every tick. Peak load hits ~2 million events/minute. A Guava in‑memory cache plus reducing punctuate to 100 ms cuts extra processing latency to 30–40 ms. Joining location back into user streams causes huge state retention and cluster churn, so the pipeline writes enriched events directly and uses a circuit breaker to forward originals on failures.
AI-written summary · about
chapters (24)
- 0:00Intro
- 3:16THE PIPELINE
- 5:01TARGETED ADVERTISING IN SCHIESTED
- 6:54THE LOCATION API
- 8:19REQUIREMENTS
- 9:36LOCATION API CHALLENGES
- 12:57KAFKA CONSUMER GROUPS
- 15:18ADDING LOCATION DATA
- 15:47SETTING UP THE APPLICATION
- 16:52READING THE INPUT #1
- 18:55READING THE INPUT #2
- 20:03TO REQUEST OBJECTS
- 21:17PRODUCING THE OUTPUT
- 22:21RECAP: KAFKA STREAMS DSL
- 22:43PROCESSOR API
- 23:23LOCATION API TRANSFORMER
- 26:42JOINING THE STREAMS
- 28:01JOIN WINDOWS
- 29:25THE LOCATION TRANSFORMER
- 29:36LOCATION TRANSFORMER PERFORMANCE
- 30:43THE JOIN COMPONENT
- 34:42JOIN PERFORMANCE
- 35:42AVERAGE CLUSTER SIZES
- 37:21CONCLUSION
related talks
32mScaling your Kafka streaming pipeline....
36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzz
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute
32mShould you read Kafka as a stream or in batch? Should you even care?
27mUsing Akka Streams for Web scale Data Ingestion
32mDetecting Events on the Web with Java, Kafka and ZooKeeper
31mBridge to the Cloud
38mData Pipeline with Apache Kafka