A Gentle Introduction to Building Data-Intensive Applications
summary
Claims how to handle exploding data (80 ZB in 2022 to ~175 ZB by 2025) and demonstrates a concrete pipeline: consuming a Redpanda stream, ingesting via SingleStore Pipelines, transforming with Stored Procedures in Go, and surfacing results on a Grafana dashboard. Architecture focuses on reliability (replicated leaf nodes), scalability (horizontal sharding and fan‑out vs join tradeoffs with Twitter numbers: thousands to hundreds of thousands of tweets/sec), and maintainability; demos and code examples aim to show the pipeline working end-to-end.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:13Data-intensive apps
- 3:40Scalable systems tenants
- 5:27Why data intensity matters
- 6:52Industry use cases
- 7:56Architecture challenges
- 9:20Reliability design
- 13:32Scaling under load
- 17:25Long-term maintainability
- 20:51Recap and future-proofing
- 21:55Next steps and resources
related talks
41m7 steps to build your own data pipeline
30mFive Challenges in Real-time Stream Processing and Five Solutions
41mOn Data: Sky above, sand below, peace within
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
34mTales of Data Architecture Evolution
05mThe Future of Data Warehousing
32mBuilding a scalable data platform
44mProcess and analyze millions of events per second with open source tools