Min and Max Aggregations with Updates in Real Time

on YouTube ↗20m

" summary

Real-time ingestion uses a Kafka Streams app to compute count, sum and newly added min/max over roll-ups (minute/hour/day) while supporting record updates; the implementation keeps per-key heaps (uses a min-heap for max and vice versa), stores heap state in the changelog Avro record, and ships aggregated rows to Apache Kudu. Pipeline handles ~20,000 records/sec with query availability under two minutes (usually seconds). Edge case: a bounded heap can evict values and produce incorrect mins/maxes; batch Spark fixes older (>3 days) updates.

AI-written summary · about

" chapters (6)

1. 0:00Introduction
2. 1:06Real-time analytics architecture
3. 4:09Data set and cube setup
4. 6:36Why math.max fails
5. 9:10Heap-based min-max logic
6. 12:00Q&A session

" related talks

• 19mProcessing High Volume Updates in HiveOwen O'Malley · Berlin Buzzwords 2012
• 41mFrom Batch to Realtime with HadoopLars George · Berlin Buzzwords 2012
• 36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzzAmeya Kanitkar · Berlin Buzzwords 2015
• 08mReal-Time Data Systems: Speed, Scale, and ResilienceLakshmi Bhargavi Mullapudi · Conf42 JavaScript 2025
• 11mBig Data Analytics: Immutability & IdempotenceKnut Nesheim · GOTO Berlin 2014
• 27mUsing Akka Streams for Web scale Data IngestionJie Ren, Greg Methvin · Scale By The Bay 2018
• 22mHow We Built SQL Rollups on Streaming DataKaren Li, Tudor Bosman · Scale By the Bay 2021
• 38mReal-time analytics with Apache Cassandra & Apache SparkChristopher Batey · Berlin Buzzwords 2015