Scale By the Bay 2021 intro video by Ido Nadler
summary
Conference intro states a production data stack that runs 250 Billion events/day on AWS and describes the actual pipelines and infra built to do it. It lists concrete tools — Kafka, Spark, Airflow, Kubernetes — and reports deployment patterns, failure modes observed, latency and cost numbers, and what engineering changes improved throughput. The piece names specific artifacts, how they are wired together, and whether the resulting system stays performant and reliable under heavy load.
AI-written summary · about
related talks
20mScale By The Bay 2019: Sharanya Santhanam & Shiv Toolsidass
04mInterview
44mProcess and analyze millions of events per second with open source tools
07mFrom Scaling APIs to Scaling Data to Scaling Your Team
04mInterview
05mInterview
35mStreaming and Real-Time Analytics
39mLarge scale streaming analytics