Billing the cloud: Real world stream processing

on YouTube ↗40m

" summary

Naive cron jobs prove simple but fail under scale, so the billing pipeline uses Kafka stream topics to partition by account, per-minute IP usage and event messages for VM create/scale/stop, and a hybrid approach: consumers keep in-memory state and apply hourly inventory snapshots for reconciliation. Usage records commit only after billing writes, via manual offset commits, preventing overbilling while tolerating short gaps; system runs with low ops cost and already powers DNS and log pipelines.

AI-written summary · about

" chapters (10)

1. 0:00Start
2. 2:40Business Model
3. 10:51Drawbacks
4. 17:47Operational simplicity
5. 18:37Integration
6. 27:06Process crashes
7. 27:30Reconciliation
8. 28:52Avoiding double billing
9. 32:19Looking back
10. 33:28What about batch

" related talks

• 31mBridge to the CloudSteve Howard · Indy Cloud Conf 2020
• 32mShould you read Kafka as a stream or in batch? Should you even care?Opher Dubrovsky, Ido Nadler · Berlin Buzzwords 2021
• 39mUsing Streaming Data Pipelines in the Cloud to Achieve SuccessJohn Lavoie · Data Con LA 2021
• 32mScaling your Kafka streaming pipeline....Opher Dubrovsky, Ido Nadler · Scale By the Bay 2021
• 22mVideo Access-Log Processing with Apache FlinkScott Kidder · Scale By The Bay 2018
• 31mHelp I need a stream processor - learning how to chose ...Andrew Psaltis · Berlin Buzzwords 2016
• 24mHow we Attributed Cost Per Tenant in a Kubernetes EnvironmentYizhar Gilboa · KubeDay Israel 2023
• 43m800 mio. IP addresses to coordinates using Kafka Streams #bbuzzHåkon Åmdal · Berlin Buzzwords 2018